Consultant Data Engineer IRC302341

Data Analytics & Business Intelligence (BI)

Experience:

5-10 years

Location:

United Kingdom - Glasgow, London

Skills:

Apache Airflow, AWS, Data Lake / Lakehouse, Databricks Certified (Data Engineer, ML), DBT, Python, SQL

We are seeking a highly skilled Сonsultant Data Engineer to be the driving force behind this massive architectural shift. You will be hands‑on in moving a highly complex, highly regulated data ecosystem from an on‑premise Teradata environment to a cutting‑edge AWS “Lakehouse” architecture.
In this role, you will focus on building robust data pipelines, modernizing legacy code, and ensuring flawless data delivery. You will work closely with System Analysts, Automation Engineers, and Data Architects to convert hundreds of legacy scripts into modern dbt models, orchestrate them via Astronomer (Airflow), and establish governed warehousing in Databricks.

Required Skills & Technologies

• Cloud Platforms:
  Hands‑on engineering experience within the AWS ecosystem (S3, IAM, Glue) and deep expertise in Databricks (SparkSQL, PySpark, governed cataloging).
• Modern Data Stack:
  Strong proficiency in dbt (Core or Cloud) for data transformation and understanding of modern table formats, specifically Apache Iceberg.
• Orchestration:
  Experience building and maintaining data pipelines using Apache Airflow (Astronomer experience is a strong plus).
• Languages:
  
  Expert‑level SQL (ability to read Teradata SQL/BTEQ and write highly optimised Spark/Databricks SQL) and strong Python skills for pipeline development and Airflow DAGs.
• Legacy Systems:
  Familiarity with Teradata architecture and utilities (Fast Export, BTEQ) to effectively understand the source systems being decommissioned.

• Experience:
  
  7+ years of dedicated Data Engineering experience, with a proven track record in complex data migration projects (On‑Premise to Cloud).
• Financial Services Background:
  Experience working in a highly regulated banking or financial services environment is strongly preferred, particularly within Risk or Finance domains.
• Engineering Best Practices:
  Strong grasp of CI/CD for data pipelines, Git version control, data modelling (Kimball/Inmon), and infrastructure as code (Terraform).
• Communication:
  Ability to work collaboratively in a fast‑paced, cross‑functional agile team, effectively communicating technical blockers to technical leads and project managers.

• Lakehouse Engineering:
  Build and maintain scalable data pipelines to migrate Consumer Risk datasets from Teradata to Amazon S3, utilising Apache Iceberg for the lakehouse layer and Databricks for governed warehousing.
• Code Modernisation (BTEQ to dbt):
  Perform complex code conversions, transforming up to 500 legacy Teradata BTEQ scripts into modular, version‑controlled, and highly performant dbt (data build tool) models.
• Pipeline Orchestration:
  Design, develop, and monitor complex Directed Acyclic Graphs (DAGs) using Astronomer (Apache Airflow) to orchestrate end‑to‑end data workflows.
• Data Lineage & Transparency:
  Integrate pipelines with Open Lineage to ensure complete, end‑to‑end data traceability, providing the transparency required for regulatory compliance.
• Reconciliation & Parity:
  Work alongside the QA/Automation team to execute automated reconciliation frameworks, debugging and resolving data discrepancies to ensure strict attribute‑level parity and timing equivalence during governed parallel runs

#LI-NV1
#LI-Hybrid

Our goal is to build an inclusive positive culture where everyone can feel comfortable being themselves, empowering people to create their own high standards and therefore more value. We work together to promote fairness while recognising, valuing and embracing differences – providing a transparent support structure and generous training budget to help our people develop skills to progress their career. Our region supports a hybrid model that can flex across a wide spectrum of working options determined by business, customer and individual needs.

You’ll benefit from a comprehensive health and wellness plan, private healthcare (clinical and mental wellbeing), discounted gym memberships, and in‑office yoga…