Principal Data Engineer

The Principal Data Engineer will be accountable for shaping the technical direction for data engineering initiatives and services within Technology & Solutions. This role combines deep technical expertise in Databricks and cloud data engineering with proven team leadership capabilities and a passion for driving innovation. The successful candidate will be accountable for delivering enterprise-scale data engineering solutions, championing modern engineering practices including AI-assisted development, and fostering a high-performing team culture focused on excellence, collaboration, and continuous growth.

Description of the role:
Data Engineering & Delivery Excellence

• Design and deliver enterprise-scale data solutions on cloud platforms (Azure preferred), leveraging Databricks as the core data engineering platform
• Build and optimize sophisticated ETL/ELT data pipelines using PySpark, SQL, and Python, orchestrating complex data workflows through Databricks Workflows, Delta Live Tables, Azure Data Factory
• Implement and maintain Delta Lake storage architectures and Unity Catalog governance frameworks, ensuring data quality, security, and compliance across the data estate
• Design data solutions for scalability, performance, resilience, and operational excellence, embedding enterprise-grade standards from inception through production deployment
• Lead technical discovery and requirements gathering with senior stakeholders, translating business needs into actionable data platform strategies and technical roadmaps
• Integrate data platforms with business intelligence and analytics tools including Databricks AI/BI and Power BI to enable self-service analytics and data-driven decision making

• Establish, evolve, and enforce data engineering standards, coding practices, and quality gates that enable safe, scalable, and maintainable data platform delivery
• Champion modern software engineering practices within data engineering contexts, including Git version control workflows, automated testing frameworks, and CI/CD deployment pipelines for data workloads
• Drive comprehensive observability, monitoring, and alerting for data pipelines and platforms, ensuring operational readiness and rapid incident response
• Promote AI-augmented data engineering practices, leveraging Git Hub Copilot, Claude Code, and other approved AI coding assistants to enhance productivity while maintaining code quality and standards

• Provide technical direction and engineering accountability for data engineering initiatives, leading teams through hands‑on contribution and solving key business challenges
• Coach and mentor data engineers at various career levels, fostering a culture of technical excellence, continuous learning, and innovation adoption
• Manage delivery of data engineering projects using Agile methodologies (Scrum, Kanban), balancing technical execution with people leadership and stakeholder management
• Foster collaboration across engineering, architecture, and product teams, breaking down silos and promoting knowledge sharing across the organization
• Represent data engineering in portfolio planning discussions, technical governance forums, and architectural review boards
  Experience in data engineering disciplines, with advanced, hands‑on expertise in Databricks platform including PySpark framework, Delta Lake storage format, and Unity Catalog governance
• Extensive experience in designing advanced data models, including medallion architectures, data lakes, and enterprise data warehouses
• Expert-level proficiency in SQL for complex data transformation, optimization, and analytical workload development
• Advanced proficiency in Python for data engineering, pipeline automation, and integration development
• Deep expertise in PySpark for data processing and transformation
• Proven mastery of cloud platforms with strong preference for Microsoft Azure
• Expertise in ETL/ELT development and data pipeline orchestration (e.g. Databricks Workflows, DLT, ADF)
• Extensive experience with Git version control, Git Hub collaboration workflows, and modern CI/CD practices including automated testing…