Data Engineer

The West of England Combined Authority are seeking a highly skilled Data Engineer to support the optimisation, documentation, and enhancement of our existing data pipelines within the Databricks Lakehouse environment. This role will involve hands‑on development, reverse‑engineering of current solutions, and close collaboration with internal teams to ensure efficient BAU operations and future scalability.

Location:West of England Combined Authority

Set-up: Hybrid – 2-3 days on-site

Rate:£249/day via umbrella inside IR35

Duration:Initial 3-month, subject to extension

Hours:Full-time position

• Develop, optimise, and debug Databricks notebooks, Delta Lake workflows, and Genie‑bot automations
• Design, maintain, and enhance ETL pipelines
  , ensuring reliability and performance
• Validate, transform, and orchestrate data using Python and SQL within the Lakehouse environment
• Manage data ingestion and scheduling processes via Azure Data Factory
• Reverse‑engineer, document, and communicate existing data flows and pipeline structures for handover to internal teams

• Strong hands‑on experience with Databricks
  , including notebooks, Delta Lake, and Genie bots
• Proven background in designing and maintaining ETL pipelines
  , including debugging and optimisation
• Advanced Python skills for transformation, automation, and pipeline development
• Solid SQL proficiency for querying, validation, and performance tuning
• Experience using Git Hub for version control, collaboration, and code management