Data Engineer

Role Overview

Data Engineer at The Institute of Foundation Models (IFM). This role specializes in Natural Language Processing (NLP) and large-scale data processing.

Company Overview

The Institute of Foundation Models is a dedicated research lab for building, understanding, using, and risk-managing foundation models. The mandate is to advance research, nurture the next generation of AI builders, and drive transformative contributions to a knowledge-driven economy.

Role Purpose

Gather, curate, and prepare high-quality datasets to support cutting-edge NLP research. Enable researchers by delivering essential data through efficient and scalable engineering practices, positioning the institute as a global hub for high-performance computing in deep learning.

Key Responsibilities

Data Pipeline Development

• Develop and maintain robust data pipelines to support NLP research at scale.
• Implement efficient and scalable engineering practices for data processing.

Data Acquisition and Preparation

• Conduct web crawling to gather large-scale datasets.
• Refine LLM-generated content for research use.
• Curate and prepare high-quality datasets that meet research requirements.

Skills & Competencies

• Proficiency in Python and related technologies.
• Expertise in Natural Language Processing (NLP) and large-scale data processing.
• Web crawling and data acquisition techniques.
• Data pipeline design and implementation.

Additional Information

• Work alongside world-class researchers, data scientists, and engineers on cutting-edge foundation model training.
• Participate in the development of groundbreaking AI solutions with potential to reshape entire industries.
• Contribute to establishing high-performance computing capabilities in deep learning.