More jobs:
Job Description & How to Apply Below
Role Overview We are looking for a highly experienced Senior Data Engineer to lead the re-engineering of an existing enterprise data platform built on Azure Synapse Analytics. The role requires deep technical seniority to audit, understand, and validate a complex end-to-end data architecture spanning source ingestion through to consumption — and to drive a future migration of validated workloads to Databricks.
This is not a greenfield role: it demands the ability to reverse-engineer existing implementations, assess their correctness, and own the technical migration strategy.
Key Responsibilities
Lead the technical assessment and re-engineering of an existing enterprise data platform, spanning all layers from source ingestion through to data consumption
Reverse-engineer, document, and validate existing pipeline logic, data models, transformation frameworks, and data governance controls
Identify gaps, defects, and technical debt across the platform and remediate where implementations are incorrect or sub-optimal
Ensure correctness of data processing patterns including change data capture, slowly changing dimensions, deduplication, and business reconciliation
Design and implement target-state architectures aligned to modern lakehouse principles, ensuring feature parity and business logic fidelity during transitions
Manage platform evolution initiatives, including parallel-run phases where multiple implementations operate simultaneously, validating output consistency before cutover
Define and execute migration strategies for existing workloads to modern data platforms, preserving existing governance and control framework semantics
Re-implement ingestion, transformation, and orchestration pipelines on target platforms, maintaining audit, quality, and reconciliation standards
Collaborate with business, data governance, and architecture stakeholders to validate embedded business rules and data quality requirements
Provide technical leadership across re-engineering and migration work streams, contributing to decommission planning for legacy components
Core Technical Skills
Azure Synapse & Data Platform Mandatory hands-on expertise with:
Azure Synapse Analytics (Pipelines, Spark Pool, Dedicated SQL Pool)
Azure Data Lake Storage Gen2 (ADLS Gen2)
Delta Lake on Azure (Synapse Lakehouse patterns)
Oracle Golden Gate Replication for real-time source integration
Azure Analysis Services and Power BI consumption layer patterns
Deep understanding of medallion architecture:
Raw / Harmonized / Conformed / Consumption layers
Strong knowledge of SCD Type 0/1/2, CDC patterns, soft/hard delete, and retroactive change processing
Experience with Synapse SQL Pool — stored procedures, control tables, and data quality validation patterns
Experience with audit, balance, and control frameworks — parameterized, modular pipeline governance at enterprise scale
Familiarity with config-driven and automation-first pipeline patterns (YAML, PySpark, SQL-driven generation from mapping documents)
Databricks & Lakehouse
Hands-on experience with Azure Databricks (Delta Live Tables, Unity Catalog preferred)
Strong Apache Spark skills (PySpark / Spark SQL)
Experience migrating workloads from legacy data warehouse or Synapse environments to a Databricks Lakehouse
Ability to re-implement governance and control frameworks natively in Databricks (audit logging, reconciliation, DQ checks)
Experience with Delta Lake features: MERGE, CDC, time travel, schema enforcement
Data Engineering & Development
Strong Python and SQL programming skills
Experience with ETL/ELT at scale: denormalization, surrogate keys, directory tables, curated data models
Experience integrating complex data sources:
Oracle DB, SQL Server, Azure SQL DB, file systems, Salesforce, APIs
Strong data modelling skills: relational, dimensional, and lakehouse-oriented
Dev Ops & Automation
CI/CD pipelines for data engineering (Azure Dev Ops / Git Hub Actions)
Infrastructure as Code (Terraform or ARM)
Containerization (Docker)
Experience with automated testing frameworks for data pipelines (unit testing,…
Position Requirements
10+ Years
work experience
Note that applications are not being accepted from your jurisdiction for this job currently via this jobsite. Candidate preferences are the decision of the Employer or Recruiting Agent, and are controlled by them alone.
To Search, View & Apply for jobs on this site that accept applications from your location or country, tap here to make a Search:
To Search, View & Apply for jobs on this site that accept applications from your location or country, tap here to make a Search:
Search for further Jobs Here:
×