Senior Data Engineer

About the Role

Senior Data Engineer for a public sector IT consultancy. In this role, you will develop, operate, and maintain data platforms and services that are high‑quality, cost‑efficient, and tailored to client needs, providing maximum value to analysts, scientists, and business stakeholders.

• Support clients as a senior contributor for projects, focusing on delivering engineering work and upskilling members of the client team.
• Take on a technical architect role to identify growth opportunities within the account.
• Deliver outcomes for users, ensuring alignment between operational and analytical aspects of the engineering solution.

• Enthusiasm for learning and self‑development.
• Proficiency in Git (incl. Git Hub Actions) and an ability to explain branch strategies.
• Gathering and meeting client and user requirements on a data project.
• Strong experience in IaC and guidance on deploying infrastructure to different environments.
• Ownership of cloud infrastructure underpinning data systems through a Dev Ops approach.
• Expertise in handling and transforming data types (JSON, CSV, etc.) with Apache Spark, Databricks or Hadoop.
• Understanding of modern data system architectures (Data Warehouse, Data Lakes, Data Meshes) and their use cases.
• Creating data pipelines on cloud environments, integrating error handling, and designing reusable libraries.
• Documenting and presenting end‑to‑end data processing systems with diagramming concepts such as C4 or UML.
• Guidance on implementing a robust Dev Ops approach, including Data Ops tools for orchestration, integration, and analytics.
• Experience improving resilience through vulnerability checks and appropriate testing strategies (unit, integration, data quality).
• Knowledge of SOLID, DRY and TDD principles and their practical implementation.
• Agile practices such as Scrum, XP and/or Kanban.
• Designing and implementing efficient data transformation processes at scale, for both batch and streaming use cases.
• People skills: mentoring, supportive team player, line‑management duties.
• Commercial mindset to grow accounts organically with senior stakeholders.

• Experience at a technology consultancy.
• Experience with Docker and virtual environments in the dev and CI/CD pipeline.
• Engaging senior stakeholders to gather and maintain requirements.
• Pair or mob programming with engineering teams.
• Product ionising advanced data deliverables such as machine learning models.
• Working knowledge of statistics.
• Collaboration with multidisciplinary digital and technology teams.
• Experience with in the public sector.

An increasing number of customers require a minimum SC (security check) clearance. All successful candidates for this role must be eligible for SC, which requires five years of continuous UK residency and five years of employment history (or full‑time education). If at any point it becomes apparent that you are not eligible, we will not be able to progress your application.

Actual pay will be based on skills and experience – discuss with your recruiter. Direct message the job poster from S.H.I.F.T for more details.

Mid‑Senior level

Full‑time

Information Technology

IT Services and IT Consulting