Data Engineer; Databricks

A leading global data, analytics and AI organisation is seeking an experienced Senior Azure Databricks Data Engineer to join a growing team delivering large-scale cloud data transformation programmes across its Retail portfolio.

About the Role

This is an opportunity to work with modern Azure and Databricks technologies, helping build scalable data platforms that support customer analytics, product insights, supply chain optimisation and business-critical reporting. You'll join a highly collaborative data engineering function working on enterprise-scale platforms, designing data pipelines, Lakehouse architectures and modern cloud-native analytics solutions. You will be responsible for designing, building and supporting scalable cloud data solutions using Azure technologies and Azure Databricks.

Working alongside Architects, Data Scientists, Analysts and Business Stakeholders, you will help deliver high-quality data products while driving engineering best practices, performance optimisation and platform scalability.

Responsibilities

• Key Responsibilities
• Design, develop and maintain enterprise-scale Azure Databricks solutions.
• Build and optimise scalable ETL/ELT pipelines using Azure Data Factory and Databricks.
• Process and transform large datasets using Python, PySpark and Spark SQL.
• Develop batch and near real-time data ingestion solutions.
• Implement robust error handling, validation and monitoring frameworks.
• Design and implement modern Data Lake and Lakehouse architectures.
• Develop scalable dimensional data models supporting analytics and reporting.
• Build and maintain enterprise Data Warehouses and analytical data platforms.
• Contribute to architecture decisions and platform modernisation initiatives.
• Promote engineering best practices across performance, scalability and maintainability.
• Analytics & Business Collaboration
• Translate business requirements into scalable technical solutions.
• Support customer, product, sales, inventory and supply chain analytics initiatives.
• Work closely with stakeholders to deliver trusted data products and reporting solutions.
• Support self-service analytics and downstream BI environments.
• Dev Ops & Governance
• Implement CI/CD pipelines and Infrastructure-as-Code best practices.
• Manage source control using Git and Azure Dev Ops.
• Ensure data quality, governance and security standards are maintained.
• Produce technical documentation, architecture designs and support material.

Required Skills

• 8+ years' experience within Data Engineering environments.
• Strong experience with Azure Databricks.
• Hands-on development experience using Python and PySpark.
• Experience with Azure Synapse Analytics and Azure Data Lake Storage.
• Strong SQL development and query optimisation skills.
• Experience building and supporting ETL/ELT frameworks.
• Data Warehousing and dimensional modelling experience.
• Experience working with enterprise-scale cloud data platforms.
• Version control and CI/CD experience using Git and Azure Dev Ops.
• Strong stakeholder engagement and requirements gathering skills.
• Experience working within Agile delivery environments.

Preferred Skills

• Retail, Ecommerce or Consumer industry experience.
• Exposure to Microsoft Fabric.
• Experience working with streaming technologies such as Kafka.
• Experience with Databricks Delta Lake and Lakehouse architecture.
• Knowledge of Data Vault modelling techniques.
• Exposure to Snowflake or other modern cloud data platforms.
• Terraform or Infrastructure as Code experience.
• Experience supporting Data Governance and Data Quality initiatives.
• What We're Looking For
• Strong problem-solving and analytical mindset.
• Passion for modern data engineering and cloud technologies.
• Comfortable working in a highly collaborative environment.
• Ability to take ownership of solutions from design through to deployment.
• Someone who enjoys building scalable platforms rather than simply maintaining existing ones.
• Strong communication skills with both technical and non-technical stakeholders.