Senior Data Engineer

Senior Data Engineer - UAE

Position Overview

Location:

Abu Dhabi, UAE

Work Requirement:
Client requirement to have people onsite with office visits.

• Implementing Databricks pipelines using Delta Live Tables (declarative pipelines).
• Rewriting proprietary Foundry transforms into optimised PySpark on Databricks.
• Moving from Foundry's build system to modern CI/CD (IaC-managed Databricks work spaces).
• Performing performance tuning on Databricks Photon to match or exceed Foundry interactivity.
• Contributing to the reconciliation engine (Palantir vs Databricks output equivalence).
• Managing pipeline rebuild, validation, internal testing, and UAT preparation.

• Proven production-level Databricks experience.
• Strong hands-on coding capability.
• Experience building and optimizing modern data pipelines.
• Ability to work within pod-based delivery teams.
• Experience owning pipeline migration and rebuild activities from design through validation.
• Databricks Databricks hands-on experience.
• Delta Live Tables.
• Unity Catalog.
• Workflows.
• Production experience required (not lab-based experience).
• Data Engineering & Development PySpark.
• Advanced SQL.
• Data modelling.
• Schema design.
• Medallion architecture.
• Lakehouse architecture.
• Dev Ops & Deployment Modern CI/CD for data pipelines.
• Infrastructure as Code (IaC).
• Git-based deployment.
• Communication Strong written and verbal English skills.
• Ability to interact directly with Databricks Professional Services (DBX).

• Certifications Databricks Certified Data Engineer Professional.
• Associate certification at minimum.
• Domain Experience Financial services.
• Audit.
• Asset management.
• Platform Experience Prior Palantir Foundry exposure, including:
  Code Repo.
• Code Workbook.
• Pipeline Builder.
• Migration Experience.
• Experience with migrations from legacy platforms such as:
  Informatica.
• Hyperion.
• Oracle.
• Palantir Foundry.
• Databricks.

• Candidates with any of the following experience will be advantageous:
  Azure Data Factory (ADF).
• Azure Synapse.
• dbt.
• Airflow.
• Equivalent orchestration tools.

Note:

Client estate is Azure-heavy.