Lead​/Principle Data Engineer

Data Engineer

Clearance Required:

Active SC Clearance (Mandatory)
The Opportunity
We're looking for a talented  Data Engineer  to join a growing team delivering secure, large-scale data solutions for government and public sector customers.
This is an exciting opportunity to work on complex data platforms that support critical decision-making and operational outcomes. You'll play a key role in building and enhancing secure batch and real-time data processing capabilities, working with modern cloud technologies and industry-leading open-source tools.
As part of a collaborative Agile team, you'll help design, develop, and optimise data pipelines, ensuring data is accurately processed, transformed, and delivered to downstream consumers.

This role offers exposure to cutting-edge cloud and data technologies alongside the opportunity to work on meaningful projects within highly secure environments.

Please note:

Active SC Clearance is an essential requirement for this position. Due to the nature of the work, candidates must already hold current and active SC Clearance. Unfortunately, sponsorship or clearance applications cannot be supported for this role.

Key Responsibilities
Develop, test, and maintain robust data pipelines that meet both functional and non-functional business requirements.
Design, build, and optimise data processing solutions to support batch and real-time analytics capabilities.
Contribute to logical and physical database design activities across a range of data platforms.
Expand and enhance platform capabilities, enabling the business to solve new and evolving data challenges.
Analyse complex datasets and identify opportunities to improve data processing, quality, and performance.
Ensure data models align with established data architecture standards, maintaining consistency across entities, relationships, and definitions.
Perform root cause analysis on data-related issues, identifying and implementing effective solutions.
Collaborate with engineers, architects, and stakeholders to deliver scalable and reliable data solutions.
Support continuous improvement initiatives, helping to drive best practices across data engineering and platform development.
Produce clear technical documentation covering data pipelines, processes, and system enhancements.
Skills & Experience
Essential
Experience developing and maintaining  data pipelines  using modern data processing technologies and frameworks.
Knowledge of  Agile  delivery methodologies and experience working within fast-paced development environments.
Strong understanding of  data modelling , data structures, and their advantages and limitations across different use cases.
Experience working with  public cloud platforms , particularly  AWS , including services such as  EC2, RDS, Lambda, Athena, and Glue .
Experience configuring and optimising  Relational  and  No

SQL  databases, including query tuning and performance optimisation.
Proficiency in programming and scripting languages such as  Python .
Experience applying  Test-Driven Development (TDD)  methodologies and associated testing frameworks.
Strong analytical and problem-solving skills with the ability to work through complex data challenges.
Excellent communication skills and the ability to work effectively within multidisciplinary teams.
Bachelor's degree or equivalent commercial experience in a relevant discipline.
Desirable

Experience with CI/CD tooling such as  Jenkins  or  Drone .
Knowledge of configuration management tools including  Ansible  and  Packer .
Experience using  Terraform  for Infrastructure-as-Code deployments.
Understanding of both  Linux  and  Windows  environments.
Exposure to modern cloud-native architectures and Dev Ops practices.
Experience working within government, defence, or other highly regulated environments.
What You'll Bring
A passion for data engineering and solving complex technical problems.
A strong attention to detail and commitment to delivering high-quality solutions.
A collaborative mindset and the ability to work effectively within Agile teams.
A proactive approach to learning new technologies and expanding your technical skillset.
The ability to translate business requirements into scalable and efficient data solutions.
A desire to contribute to projects that deliver meaningful impact within secure and mission-critical environments.
Security Clearance -   Mandatory Requirement
Candidates must hold current and active SC (Security Check) Clearance at the time of application.
Applications from candidates who…