Senior Data Engineer; Databricks

Senior Data Engineer

Experience: 5 Years

Location: Abu Dhabi

We are seeking an experienced and proactive Data Engineer to lead the development of scalable production‑grade data pipelines while mentoring junior engineers and shaping engineering standards. You will work hands‑on with Databricks, including Azure Databricks, Delta Lake and Apache Spark, to support data‑driven decision‑making across analytics, product, and data science teams. This is a role that combines technical depth with team leadership, ideal for someone ready to take ownership of critical data systems while guiding others.

• Design, develop, and optimize scalable data pipelines using Databricks and Azure Databricks, including Apache Spark and Delta Lake.
• Lead and mentor junior engineers through code reviews, architecture discussions, and technical guidance.
• Collaborate with cross‑functional teams to define data requirements and deliver reliable data solutions.
• Manage workflows across Azure Data Lake Storage, Azure Synapse Analytics, and Azure Data Factory or equivalent orchestration tools.
• Contribute to platform architecture decisions involving data modeling, processing frameworks, and performance tuning.
• Implement CI/CD pipelines for data workflows using tools like Azure Dev Ops or Git Hub Actions.
• Enforce engineering best practices around testing, documentation, and maintainability.
• Monitor, troubleshoot, and resolve issues related to data pipeline performance and reliability.
• Ensure data governance, quality, and compliance by integrating with tools like Unity Catalog or Azure Purview.

• Bachelor’s degree in computer science or a related field.
• 5 years of experience in data engineering or a closely related technical field.
• Strong proficiency in Python and SQL for building and managing data workflows.
• Hands‑on experience with both Databricks and Azure Databricks, including production use of Apache Spark and Delta Lake.
• Familiarity with cloud‑based data platforms, preferably Microsoft Azure.
• Experience with Azure Data Factory, Azure Data Lake, and/or Azure Synapse Analytics.
• Demonstrated ability to lead technical projects and mentor junior engineers.
• Solid grasp of CI/CD practices and version control systems (e.g., Git).

• Exposure to data transformation tools like dbt.
• Familiarity with data governance tools such as Unity Catalog, Azure Purview, or similar.
• Experience with containerization (e.g., Docker) and infrastructure‑as‑code tools (e.g., Terraform, Bicep).
• Strong communication and documentation skills; able to interface with technical and non‑technical stakeholders.