Databricks Engineer

Role:
Databricks Engineer

Location:
London (3 days in a week)

Employment:
Fulltime

We are seeking an experienced Azure Databricks Data Engineer to design, develop, deploy, and optimize scalable data solutions on Microsoft Azure. The ideal candidate will bring strong hands-on expertise in Azure Databricks, Apache Spark, PySpark, Python, and SQL, with proven experience building production-grade data pipelines, implementing governance through Unity Catalog, and automating deployments using Git Lab-based CI/CD and Databricks Asset Bundles.

• Design, build, test, and maintain scalable batch and streaming data pipelines using Azure Databricks, Apache Spark, PySpark, Python, SQL, and Delta Lake.

• Develop reusable ETL and ELT frameworks for data ingestion, transformation, validation, and publishing across lakehouse layers.

• Design and manage Delta Tables, including schema evolution, data quality controls, reliability, and performance optimization.

• Implement data governance, access control, cataloging, and lineage standards using Unity Catalog.

• Create, schedule, monitor, and troubleshoot production workloads using Databricks Jobs and workflows.

• Package and deploy Databricks resources across environments using Databricks Asset Bundles and Git Lab-based CI/CD pipelines.

• Integrate Databricks with Azure Data Lake Storage and Azure Data Factory for secure, reliable data processing.

• Optimize Spark workloads, clusters, jobs, and storage patterns for performance, scalability, reliability, and cost efficiency.

• Apply coding standards, version control, testing, documentation, and operational best practices using Git Lab and Azure Dev Ops.

• Collaborate with architects, analysts, and engineering teams to translate business requirements into maintainable technical solutions.

• Strong hands-on experience with Azure Databricks and the Apache Spark execution architecture.

• Advanced proficiency in PySpark, Python, and SQL for large-scale data processing and transformation.

• Practical experience with Delta Lake, Delta Tables, Unity Catalog, Databricks Jobs, and Databricks Asset Bundles.

• Proven experience designing and operating production-grade ETL or ELT pipelines on Azure.

• Hands-on experience implementing CI/CD pipelines using Git Lab, including automated validation and multi-environment deployments.

• Working knowledge of Azure Data Lake Storage, Azure Data Factory, and Azure Dev Ops.

• Demonstrated ability to tune Spark workloads and troubleshoot data pipeline performance and production issues.

• Strong understanding of data engineering, governance, security, version control, testing, and deployment best practices.

• Databricks Certified Data Engineer Associate or Professional certification.

• Microsoft Azure data engineering certification or equivalent cloud certification.

• Experience with medallion architecture, data quality frameworks, streaming pipelines, infrastructure as code, or lakehouse monitoring.

• Exposure to enterprise data governance, regulated environments, or large-scale cloud data modernization programs.