Senior Data Platform Engineer

Role Overview

Senior Data Platform Engineer at BlackStone eIT. This role leads the design, development, and maintenance of enterprise data platforms that are scalable, reliable, and optimized to support advanced analytics and business intelligence efforts.

Role Purpose

To design, build, and maintain scalable data platforms that facilitate seamless data ingestion, processing, storage, and accessibility while ensuring data infrastructure meets enterprise security, governance, and compliance requirements.

Key Responsibilities

Platform Architecture & Design

• Design, build, and maintain scalable data platforms to support diverse data workloads.
• Own pipeline design patterns, orchestration strategy, infrastructure-as-code, and monitoring stack.
• Translate business requirements into technical solutions in collaboration with cross-functional teams.

Data Integration & Extraction

• Develop and optimize ETL/ELT pipelines to ensure efficient data movement and transformation.
• Own enterprise data extraction approach and build the first pipelines yourself.
• Extract data from SAP FI/CO (budget, actuals, commitments via OData/RFC).
• Extract data from Microsoft Graph API (SharePoint, M365).
• Extract data from SuccessFactors OData API.

Data Staging & Transformation

• Design staging layer architecture including PostgreSQL staging schema.
• Implement raw-to-clean transformation patterns and refresh strategies.
• Apply transformation frameworks like dbt or equivalent for transformation and quality testing.

Data Quality & Governance

• Define and enforce data quality checks and determine what pauses a pipeline versus what triggers a warning.
• Embed automated classification tagging into every pipeline in collaboration with the Governance Lead.
• Ensure data platform security, governance, and compliance with relevant regulations.

Observability & Performance

• Establish logging standards, alerting on failures, and monitoring dashboards for pipeline health.
• Monitor platform performance, troubleshoot issues, and implement improvements.
• Stay updated with the latest trends and technologies in data platform engineering.

Technical Leadership

• Provide technical leadership and mentorship to junior data engineers.
• Set standards and review work for team members.

Qualifications & Experience

• Bachelor's or Master's degree in Computer Science, Information Technology, or related field.
• 8+ years of experience in data engineering, with a focus on data platform architecture and development.
• Hands-on experience with cloud data platforms such as AWS, Azure, or Google Cloud.
• Experience with big data technologies like Hadoop, Spark, and data warehousing solutions.
• Knowledge of data governance, security, and compliance standards.
• Proven ability to lead technical projects and mentor team members.

Skills & Competencies

• SAP data extraction: OData services, RFC/BAPI, SAP Data Services, or CDS views.
• Strong proficiency in SQL and experience with relational and NoSQL databases.
• Python: async data pipelines, background jobs, scheduled tasks.
• PostgreSQL: schema design, migrations (Alembic), query optimization, partitioning.
• Azure Data Lake Storage + Synapse Analytics.
• Microsoft Graph API: SharePoint, M365, organizational data.
• Apache Airflow or Azure Data Factory.
• dbt or equivalent for transformation and quality testing.
• Redis: queue management, TTL, cache invalidation.
• Observability: structured logging, Azure Monitor or Prometheus/Grafana.
• Docker: containerized pipeline jobs.
• Infrastructure-as-code (Terraform or equivalent).
• Excellent problem-solving and communication skills.

Additional Information

• Paid Time Off.
• Performance Bonus.
• Training & Development.