Lead Data Engineer

As a Lead Data Engineer, you will take ownership of the design, architecture, and evolution of our modern data platform. You will play a key role in defining the technical direction of the data engineering function while remaining hands-on in building and scaling our core data infrastructure.

You will work closely with engineers, product managers, and regulatory experts to design scalable solutions that power our client-facing regulatory reporting platform. This role combines hands-on engineering, technical leadership, and mentorship, helping guide the team as we rebuild our data platform from legacy Python scripts and MySQL into a modern, scalable architecture using Databricks, Snowflake, Kafka, Python, and PySpark.

Our data systems are client-facing and used directly by major financial institutions, meaning the systems you design and build will power regulatory reporting and data quality across global financial markets.

What You'll Do

Technical Leadership & Architecture

• Own the architecture and evolution of the modern data platform.
• Define engineering standards and best practices for data engineering.
• Lead technical design discussions and architecture decisions across the data platform.
• Guide the team in building scalable, reliable, and auditable data systems.

Platform Development

• Design and evolve a configuration-driven data platform enabling non-engineers to define regulatory logic.
• Build scalable data pipelines processing regulatory transaction data using Databricks, Snowflake, Kafka, Python, and PySpark.
• Develop robust data quality frameworks including testing, validation, monitoring, and alerting.
• Ensure pipelines are auditable, reproducible, and resilient, supporting regulatory-grade data integrity.
• Provide technical leadership and mentorship to data engineers.
• Support the growth and development of the data engineering team.
• Conduct code reviews and promote strong engineering practices across the team.
• Help shape the future structure and capabilities of the data engineering function as the company scales.

Cross-Functional Collaboration

• Work closely with Product, Engineering, and Regulatory Insights teams to translate complex regulatory requirements into scalable data solutions.
• Partner with Dev Ops teams to optimise infrastructure for Snowflake and Databricks workloads.
• Contribute to improving development processes, platform reliability, and engineering productivity.

Must-Haves

• Strong experience building scalable data platforms and data pipelines using Python and PySpark.
• Experience designing distributed data processing systems and tuning Spark workloads for performance.
• Experience leading technical architecture decisions for modern data platforms.
• Strong SQL expertise with the ability to optimise complex queries on large datasets.
• Strong experience with data modelling and database design.
• Proven experience mentoring or leading data engineers.
• Experience building auditable and reproducible data pipelines with lineage, idempotency, and replay capabilities in regulated or high-integrity environments.
• Strong software engineering practices including clean code, testing, CI/CD, and observability.
• Ability to translate complex regulatory or business requirements into scalable technical solutions.

Nice-to-Haves

• Experience with Kafka or other event streaming platforms.
• Hands-on experience with AWS cloud infrastructure.
• Experience working in Fin Tech or financial services environments.