Databricks Technical Lead; Azure | Commodity Trading | Greenfield Data Platform

Overview

Databricks Technical Lead (Azure) | Commodity Trading | Greenfield Data Platform | Up to £1,000 per day Inside IR35 | 6-month rolling | Hybrid 2–3 days London

A Databricks Technical Lead is required for a large, complex commodity trading organisation undertaking a greenfield implementation of Databricks on Azure
. This is a strategic data platform programme, focused on building a modern, scalable lakehouse architecture to support analytics, reporting, and downstream data use cases.

This Databricks Technical Lead role will take end-to-end technical ownership of the Databricks platform, leading design decisions, setting engineering standards, and guiding delivery across a hybrid on-prem and Azure environment
.

You will act as the senior technical authority, working closely with data engineers, architects, platform teams, and senior stakeholders to ensure the platform is built correctly from day one.

Key Responsibilities of the Databricks Technical Lead:

• Lead the greenfield design and implementation of Databricks on Azure.
• Define and own platform architecture, patterns, and best practices.
• Design and oversee development of scalable data pipelines using Spark, Python, and SQL.
• Lead ingestion from on-prem data sources into Azure and Databricks.
• Drive adoption of lakehouse architecture, data modelling standards, and performance optimisation.
• Act as the technical escalation point for Databricks, Spark, and Azure data services.
• Provide technical leadership and mentorship to data engineers.
• Conduct design reviews and ensure high engineering quality.
• Work with security and platform teams on governance, access control, and compliance.
• Support early POCs, MVPs, and longer-term platform evolution.

Hybrid working: 2–3 days per week, City of London

• Proven experience as a Databricks Technical Lead or Senior Data Engineer / Architect
• Strong hands-on Databricks experience on Azure
• Experience leading greenfield data platform implementations
• Strong Python and SQL skills for data engineering
• Experience integrating on-prem systems with Azure
• Experience defining engineering standards and architecture
• Strong stakeholder communication and leadership skills

Domain experience (e.g. energy, commodities, finance) is NOT required — this role is focused on technical delivery and platform leadership.

If you are interested, reply with your latest CV or email me at Oscar