Lead Data Engineer

Job Title

Lead Data Engineer

The Lead Data Engineer helps architect and lead the development of enterprise-scale data platforms and advanced analytics solutions across multiple business units and subject areas. In addition, they assist the project delivery team with scoping and planning integration projects and contribute to the design, implementation, and support of the data artifacts in the data lakehouse and databases. This role combines deep technical expertise with leadership and mentoring responsibilities, driving innovation and best practices across the data engineering function.

• Support the Solution and Data Architect(s) to help lead the design, development, and optimization of scalable data platforms and analytics solutions that support global data strategy.
• Architect and implement robust, scalable, metadata-driven pipelines in Databricks using Delta Lake, Unity Catalog, and declarative workflows.
• Oversee the development and maintenance of Azure SQL database objects using advanced T-SQL for data quality and insight generation.
• Drive performance, reliability and observability of real-time data processing using Spark Structured Streaming and Change Data Feed strategies.
• Own CI/CD processes and infrastructure automation using Databricks Asset Bundles, Azure Dev Ops, and Infrastructure as Code.
• Define and enforce standards for reusable, metadata-driven integration patterns, including Unity Catalog governance, Declarative Pipeline design, and data quality across the engineering team.
• Manage and mentor a small team of data engineers, providing technical leadership, pair programming support, conducting design/code reviews, guiding technical decisions and aligning data engineering efforts with strategic goals
• Collaborate with cross-functional stakeholders to translate business and technical requirements into scalable data solutions, assisting the project delivery team to scope, plan and deliver new and enhanced data integrations
• Champion innovation, best practices, and continuous improvement in data engineering, while mentoring team members and fostering technical growth.
• Evaluate and introduce emerging technologies to enhance platform capabilities

• Extensive hands-on experience (6+ years) in cloud-native data engineering, including 3+ years with Databricks and Delta Lake in Azure, combined with good (2+ years) in a technical leadership or architecture role.
• Deep practical experience of the Databricks ecosystem including Delta Lake, Unity Catalog, Declarative Pipelines, Spark Structured Streaming, Change Data Feed, Serverless SQL, and Cluster management & optimization.
• Expert-level SQL skills, with a track record of designing performant queries, optimizing data models, and driving insight generation from large datasets.
• Proficiency in multiple programming languages used in data engineering and analytics (e.g., PySpark, Scala, R, Python), with experience mentoring others in their use.
• Deep understanding of cloud-native data architecture, including Lakehouse design principles, data integration patterns, and automation strategies.
• Proven ability to design and deliver scalable, production-grade data pipelines, with a focus on reliability, maintainability, and performance.
• Strong experience with CI/CD, version control (Git), ideally including Databricks Asset Bundles, and Infrastructure as Code, leading Dev Ops practices in data engineering teams.
• Skilled in requirements gathering and solution design, translating business needs into technical specifications and guiding teams through delivery.
• Demonstrated leadership in troubleshooting and resolving complex data and code issues, driving root cause analysis and long-term fixes.
• Excellent communication and stakeholder engagement skills, with experience presenting architectural decisions, documenting processes, and influencing cross-functional teams.
• Commitment to continuous improvement, staying current with emerging technologies and fostering a culture of learning and innovation within the team.

• Hands‑on experience with machine learning pipelines or integrating data…