وصف الوظيفة
Role Overview
Masterworks is seeking a Data Engineer to join the team at a client site. The role involves developing, operating, and maintaining the organization's data pipelines and data platform infrastructure.
Role Purpose
Ensure reliable data processing, platform availability, security, performance, and compliance with pipeline SLAs through the design, deployment, and optimization of enterprise data systems.
Key Responsibilities
Pipeline Development & Maintenance
- Design, develop, and maintain ELT/ETL pipelines to produce clean, structured, and reliable datasets.
- Develop and test data transformations using SQL, Python, and PySpark.
- Build and maintain data orchestration workflows using Apache Airflow.
Platform Administration & Operations
- Administer and maintain lakehouse, object storage, Apache Spark, and Trino environments.
- Monitor platform performance, respond to incidents, and perform system upgrades and patching.
- Manage user access, roles, permissions, and resource quotas across data platforms.
- Maintain infrastructure documentation, operational procedures, and technical runbooks.
Pipeline Monitoring & Optimization
- Monitor pipeline execution, troubleshoot processing issues, and proactively address SLA risks.
- Optimize storage configurations, including partitioning, data formats, and data retention.
- Participate in capacity planning, platform optimization, and cost-management initiatives.
CI/CD & Infrastructure
- Build and maintain CI/CD pipelines using Bitbucket and Jenkins.
Collaboration & Support
- Collaborate with Data Integration, Data Engineering, BI, and Analytics Engineering teams to support data initiatives.
- Provide coverage and support for Data Integration activities during team absences or periods of increased workload.
Qualifications & Experience
- Bachelor's degree in Computer Science, Information Systems, Software Engineering, or a related field.
- Minimum 5 years of relevant experience in Data Engineering or a related field.
- Hands-on experience with lakehouse architecture, object storage, and ELT/ETL development.
- Experience working with enterprise data platforms and supporting production data environments.
- Practical experience with Bitbucket, Jenkins, and CI/CD practices.
Skills & Competencies
- Strong proficiency in SQL, Python, PySpark, Apache Spark, Trino, and Apache Airflow.
- ELT/ETL pipeline development.
- Apache Airflow orchestration.
- Lakehouse architecture and object storage.
- Data platform administration.
- Access, role, permission, and quota management.
- Pipeline monitoring and incident resolution.
- Performance and storage optimization.
- Knowledge of access management, platform monitoring, incident response, and performance optimization.
الراتبغير مُفصح عنه من صاحب العمل
آخر موعد—