Data Engineer

Data Engineer

Contract:

6‑month contract initially. Worksite: 3 days per week onsite in Stevenage. Rate: £400–£500 per day via umbrella.

• Take full ownership of data pipeline testing strategy, defining scope, priorities, and standards.
• Design and implement a scalable Pytest‑based automation framework for ETL/data validation.
• Develop robust SQL‑based validation checks (reconciliation, duplicates, nulls, business rules).
• Automate validation of Excel outputs using Pandas and OpenPyXL, ensuring structural and data accuracy.
• Validate XML outputs using lxml and xmlschema, including schema compliance and business‑level rules.
• Own and automate API testing workflows using Postman and Newman.
• Integrate all testing workflows into Git Hub Actions, enabling CI/CD‑driven quality gates.
• Build clear, actionable reporting and logging, making failures easy to diagnose and debug.
• Collaborate closely with data engineers to improve testability, catch defects early, and accelerate resolution.
• Mentor and guide junior testers, establishing best practices, code standards, and team workflows.

• Strong hands‑on experience in testing data pipelines / ETL/ELT systems in complex data environments.
• Advanced SQL skills, with the ability to write complex queries for data validation, debugging, and reconciliation.
• Proven expertise in test automation using Python with Pytest, including building reusable frameworks (fixtures, utilities, modular design).
• Experience using Pandas and OpenPyXL for validating data outputs such as Excel files (structure, values, comparisons).
• Hands‑on experience validating XML data using lxml and xmlschema, including schema (XSD) validation and business rules.
• Solid experience in API testing using Postman and automation using Newman.
• Practical experience implementing CI/CD pipelines using Git Hub Actions for automated test execution and quality gates.
• Familiarity with Databricks or similar modern data platforms for large‑scale data processing and validation.
• Strong understanding of data quality principles, including data integrity, transformation validation, and reconciliation techniques.
• Experience in building QA processes from scratch or leading data testing initiatives, with ability to define standards and mentor others.

We are committed to fostering an inclusive, equitable and accessible workplace where everyone feels valued and supported. We welcome applications from all individuals, regardless of background or identity, and we encourage candidates who may not meet every listed requirement to still apply.