Principal Evals Engineer

As Principal Evals Engineer you will define how AI performance is measured across the organisation You will design and build the shared evaluation architecture used across multiple AI products and engineering teams This includes creating robust evaluation harnesses managing golden datasets automating grading detecting behavioural regressions and ensuring that engineering teams can make release decisions based on evidence rather than subjective judgement This is a senior individual contributor role You will remain highly hands on building frameworks running experiments and setting the technical standard for AI evaluation across the organisation

• Define the overall AI evaluation strategy and reference architecture across the engineering organisation
• Build shared evaluation frameworks harnesses and reusable libraries
• Develop and maintain golden datasets and dataset versioning strategies
• Design automated evaluation and grading systems
• Implement regression detection for non deterministic AI behaviour
• Evaluate LLM output quality hallucination grounding faithfulness and citation accuracy
• Build evaluation frameworks for RAG and retrieval systems
• Evaluate agentic systems including tool usage multi step reasoning and failure recovery
• Design and calibrate LLM as a judge systems against human evaluation
• Establish production monitoring online evaluation and drift detection
• Develop adversarial testing methodologies covering prompt injection jailbreaks and data leakage
• Integrate AI evaluation into CI CD pipelines and release processes
• Help engineering teams understand and apply rigorous evaluation standards independently
• Contribute to building the broader evaluation engineering function as the organisation scales

We are looking for a senior engineer with a strong track record of building AI or testing infrastructure at scale You should have experience owning evaluation frameworks rather than simply working inside an existing testing platform

• Staff or Principal level engineering experience
• Strong hands on Python engineering skills
• Deep experience with LLM or Generative AI evaluation
• Experience evaluating RAG retrieval or agentic systems
• Strong understanding of hallucination grounding and output quality measurement
• Experience building regression detection for non deterministic AI systems
• Practical experience with LLM as a judge techniques
• Strong understanding of rubric design and calibration against human labels
• Statistical literacy including sampling confidence intervals significance and inter rater agreement
• Strong CI CD and software quality engineering experience
• Experience building reusable frameworks or shared engineering libraries
• Hands on experience working with production AI systems

Experience with one or more of the following would be highly valuable

• Ragas
• Deep Eval
• Promptfoo
• Braintrust
• Lang Smith
• Langfuse
• Lang Chain or Lang Graph
• Vector databases such as pgvector or Qdrant
• AI red teaming or adversarial testing
• Voice or conversational AI evaluation
• Human evaluation programme design
• Data pipeline or performance testing
• Arabic language AI evaluation including golden datasets dialect coverage and right to left validation
• Experience working within regulated government or highly secure environments

The primary language is Python with additional use of Type Script and Java The wider environment includes custom evaluation harnesses LLM as a judge patterns golden datasets Langfuse Ragas Deep Eval Promptfoo Lang Chain Lang Graph Microsoft Agent Framework pgvector Qdrant Playwright Cypress Great Expectations Grafana Docker Kubernetes and Azure The team is pragmatic about tooling and is more interested in strong engineering judgement than experience with one specific framework