Azure Databricks Engineer

We are looking for an experienced Azure Data Engineering Lead with strong hands‑on expertise in Azure Databricks, Azure Data Factory, PySpark, SQL, Delta Lake, and Azure cloud data services. The candidate will be responsible for designing, developing, and optimizing scalable data pipelines and modern data platform solutions for a banking client.

The ideal candidate should have strong experience in building enterprise‑grade data solutions, handling large volumes of structured and semi‑structured data, and working in regulated environments such as banking or financial services.

The places that you work from day to day will vary according to your role, your needs, and those of the business; it will be a blend of company offices, client sites, and your home; noting that you will be unable to work at home 100% of the time.

• Design, develop, and maintain scalable data pipelines using Azure Data Factory and Azure Databricks.
• Build and optimise data processing frameworks using PySpark, Spark SQL, Delta Lake, and Databricks notebooks.
• Develop end‑to‑end data ingestion pipelines from multiple source systems including relational databases, files, APIs, and cloud storage.
• Implement batch and near‑real‑time data processing solutions on Azure.
• Work with banking datasets such as customer, account, transaction, payments, risk, compliance, regulatory, and reporting data.
• Design data models and transformation logic for curated, consumption, and reporting layers.
• Implement data Lakehouse architecture using Azure Data Lake Storage Gen2, Databricks, and Delta Lake.
• Ensure high standards of data quality, data validation, reconciliation, and auditability.
• Optimize Spark jobs and ADF pipelines for performance, reliability, and cost efficiency.
• Implement CI/CD practices for data pipelines using Azure Dev Ops or similar tools.
• Collaborate with architects, business analysts, data modelers, QA teams, and business stakeholders.
• Support production deployments, troubleshooting, root‑cause analysis, and performance tuning.
• Ensure compliance with banking security, privacy, and regulatory requirements.
• Provide technical leadership, mentoring, and guidance to junior data engineers.

• Strong hands‑on experience with Azure Databricks.
• Strong experience with Azure Data Factory pipeline development.
• Excellent programming experience in PySpark and Python.
• Strong SQL skills, including complex queries, joins, window functions, performance tuning, and stored procedures.
• Experience with Azure Data Lake Storage Gen
  2.
• Strong knowledge of Delta Lake, Lakehouse architecture, and medallion architecture.
• Experience in building ETL/ELT pipelines for large‑scale data processing.
• Experience with data ingestion from multiple sources such as:
  • SQL Server
  • Oracle
  • APIs
  • Flat files
  • Parquet / JSON / CSV
  • Streaming or event‑based sources
• Experience with job scheduling, dependency management, error handling, and retry mechanisms.
• Strong knowledge of data quality, reconciliation, and audit frameworks.
• Experience with Azure Dev Ops, Git, CI/CD pipelines, and release management.
• Good understanding of cloud security, access control, Key Vault, managed identities, and RBAC.

Capgemini is a Disability Confident Employer (Level
2) under the UK Government’s Disability Confident scheme. If you declare a disability and meet the minimum essential criteria for the role, please opt in during the application process.