AI QA Engineer

Location: Greater London

Job Title:

AI QA Engineer
Salary: £70K-£95K base+ 10% bonus + benefits

Location:

Moorgate, London – Hybrid, typically 2 days per week in the office
Work Type:
Permanent

Role:
We’re working with a fast-growing, AI-first technology business that is building an innovative product using cutting-edge AI.

As the business continues to scale, they’re looking for an AI QA Engineer to help solve one of the biggest challenges when building AI products: how do you confidently test and ship non-deterministic AI output at scale?

You’ll join a small QA function with a big influence across engineering. This isn’t a traditional QA environment where testing sits solely with the QA team. You’ll build the evaluation systems, automation and tooling that allow quality to scale across multiple engineering squads.

You’ll be joining an environment where AI is already central to the product and the way the engineering team works, giving you the opportunity to tackle real AI quality problems and influence how the function develops as the company grows.

Responsibilities:

• Enhance and maintain AI evaluation frameworks, including automated checks, LLM-as-judge scoring, rubrics and targeted human review

• Maintain versioned golden datasets to keep AI evaluation reproducible and auditable

• Track and improve AI quality across groundedness, hallucination rates, entity resolution and source quality

• Own and improve test automation and tooling across the product

• Work with Python, Playwright and pytest across the existing test infrastructure

• Investigate flaky tests and changing evaluation metrics, identifying and resolving problems at their root cause

• Work closely with engineering and product teams to improve testing practices across different squads

• Build tooling and share knowledge that enables engineers to take greater ownership of quality

Required Skills:

• Strong experience within AI QA, AI testing or LLM evaluation

• Hands-on experience evaluating and testing non-deterministic AI/LLM outputs

• Strong test automation engineering experience

• Experience with LLM evaluation techniques such as golden datasets, rubric design, LLM-as-judge or regression evaluation

• Experience with Python or another object-oriented programming language

• Experience with modern automation frameworks such as Playwright, pytest or similar

• Strong understanding of CI/CD and building automated testing as scalable infrastructure

• An analytical and sceptical approach to quality, with the ability to recognise when a successful test result doesn’t necessarily mean the output is correct

• Strong communication skills with the ability to explain complex quality problems to both technical and non-technical stakeholders

• Comfortable working autonomously while influencing and collaborating with engineers across multiple squads

• Must have the right to work in the UK – sponsorship is not available

Why should I apply?
This is an opportunity to work on AI quality problems that go far beyond traditional functional testing.

You’ll be testing non-deterministic AI systems, improving evaluation approaches and helping determine whether AI-generated outputs are genuinely reliable rather than simply relying on a passing test.

You’ll also have genuine influence. The QA function works across engineering, so the tooling, frameworks and approaches you build will help shape how multiple squads think about and deliver quality.

You’ll be joining a genuinely AI-first environment, working alongside people who are continually experimenting with how AI can improve both the product and the way they work.

Alongside a 10% bonus, you’ll receive employee shares/equity, private healthcare, life insurance, unlimited holiday and a £1,000 annual professional development fund.

Interested?
Apply for the role today or send your CV to (url removed)