Tech Lead; Azure Databricks Data Platform

Aubay UK is seeking an experienced Tech Lead (Azure Databricks Data Platform) to lead the design, development, and delivery of enterprise-scale data platform solutions leveraging the Azure Data Platform and Databricks ecosystem.

The successful candidate will be responsible for providing end-to-end technical leadership across data engineering initiatives, driving solution architecture, platform strategy, engineering standards, and delivery excellence. Working closely with business stakeholders, architects, product teams, and engineers, the Tech Lead will translate complex business requirements into scalable, high-performance data solutions that support critical business operations and analytics capabilities.

This role requires deep hands-on expertise in Azure Databricks, Structured Streaming, Delta Live Tables (DLT), PySpark, Python, and SQL, alongside strong stakeholder management and leadership skills. The ideal candidate will have a proven track record of delivering large-scale cloud data platforms, optimising platform performance and costs, and mentoring engineering teams to achieve successful business outcomes.

• Strong hands-on experience with Azure Databricks, PySpark, Python, and SQL
• Experience with in financial trading, commodities trading, or energy trading environments
• Understanding of trading data models and trading business processes
• Experience building and supporting enterprise-scale data platforms on Microsoft Azure
• Expertise in Structured Streaming and Delta Live Tables (DLT)
• Experience designing scalable, high-performance data solutions and architectures
• Strong understanding of data governance, data quality, and engineering best practices
• Experience leading technical delivery from design through to production support
• Strong stakeholder management and communication skills
• Experience leading and mentoring engineering teams

• Experience optimising Databricks performance and managing cloud platform costs
• Knowledge of Lakehouse architecture and modern data platform design patterns
• Experience working within Agile delivery environments
• Relevant Azure and/or Databricks certifications

• Lead the design, development, and implementation of scalable data platform solutions using Azure and Databricks
• Provide technical leadership across the full solution lifecycle, from architecture through delivery and production support
• Define and promote best practices for data engineering, governance, security, quality, and platform standards
• Design and optimise real-time data processing solutions using Structured Streaming and Delta Live Tables (DLT)
• Collaborate with business stakeholders to translate business requirements into technical solutions
• Drive technical decision-making and ensure alignment with platform strategy and business objectives
• Monitor and optimise platform performance, reliability, scalability, and cloud costs
• Support the development of data architecture, integration patterns, and engineering roadmaps
• Mentor Data Engineers and provide technical guidance across delivery teams
• Ensure successful delivery of business outcomes through strong ownership, accountability, and leadership
• Work closely with architects, product teams, and stakeholders to manage technical risks, dependencies, and priorities
• Promote continuous improvement and adoption of modern data engineering practices across the platform team

Aubay UK is a recognised InSourcing Partner for client-side deployment delivered across London. Our team, based in Canary Wharf, specialises in hiring IT professionals within London’s Energy and Fin Tech sectors, helping our clients to expand their operations with top-tier talent who are experts in their fields. We work exclusively with clients who are globally recognized as Energy Super Majors/Financial Services and innovative Fin Tech players.

Aubay UK is the most recently started branch of Aubay Group  Aubay Group is an international Digital Services Company, listed on a Euronext Stock Exchange, who have been operating for 25 years in the European market and working alongside some of the biggest names in…