Data Engineer

The West of England Combined Authority are seeking a highly skilled Data Engineer to support the optimisation, documentation, and enhancement of our existing data pipelines within the Databricks Lakehouse environment. This role will involve hands-on development, reverse-engineering of current solutions, and close collaboration with internal teams to ensure efficient BAU operations and future scalability.

Location: West of England Combined Authority

Set-up: Hybrid - 2-3 days on-site

Rate: £249/day via umbrella inside IR35

Duration: Initial 3-month, subject to extension

Hours: Full-time position

Responsibilities include:

• Develop, optimise, and debug Databricks notebooks, Delta Lake workflows, and Genie-bot automations
• Design, maintain, and enhance ETL pipelines, ensuring reliability and performance
• Validate, transform, and orchestrate data using Python and SQL within the Lakehouse environment
• Manage data ingestion and scheduling processes via Azure Data Factory
• Reverse-engineer, document, and communicate existing data flows and pipeline structures for handover to internal teams

Requirements:

• Strong hands-on experience with Databricks, including notebooks, Delta Lake, and Genie bots
• Proven background in designing and maintaining ETL pipelines, including debugging and optimisation
• Advanced Python skills for transformation, automation, and pipeline development
• Solid SQL proficiency for querying, validation, and performance tuning
• Experience using Git Hub for version control, collaboration, and code management

At Venn Group, we endeavour to respond to all applications However due to the volume of applications we receive, if you have not heard from us within 48 hours please assume you have been unsuccessful on this occasion