AI Harness & Evaluations Engineer

The places that you work from day to day will vary according to your role, your needs, and those of the business; it will be a blend of Company offices, client sites, and your home; noting that you will be unable to work at home 100% of the time.

If you are successfully offered this position, you will go through a series of pre-employment checks, including, identity, nationality (single or dual) or immigration status, employment history going back 3 continuous years, and unspent criminal record check (known as Disclosure and Barring Service)

We are developing a new AI-native product organisation within Capgemini Financial Services. We build products, not projects: software for insurance claims, payment operations, and health operations, sold to banks, insurers, and health plans. Three product lines run on one shared platform, built by a deliberately small, senior team. Our engineering model is agentic: engineers author the specifications, tooling, evaluation suites, and guardrails, and AI agents do most of the implementation.

Humans own every consequential decision, and in our regulated domains some decisions are human-only by design.

In a regulated domain, an AI product is only as trustworthy as the evidence that it works. You will own that evidence for one product line: the golden datasets that define correct behaviour, the gates that stop a regression reaching a client, the drift monitors that catch quality decaying in production, and the evidence pack a bank or insurer's model-risk function reads before they let us anywhere near their operations.

You will build those datasets alongside a full-time domain expert, because in claims and payments the definition of a correct answer is a professional judgment, not a label.

You also own how your line runs its harness: the steering files, specification conventions, and agent tooling your engineers work inside, adapted from the platform group's patterns to the shape of your domain.

• The line's evaluation suites: domain golden datasets that are human-anchored and domain-anchored, never derived from the system under test
• Gate design: per-cohort thresholds, statistical gating with honest sample sizes, and judge calibration with pinned versions so a gate means the same thing this month as last
• Evaluations wired into CI as merge gates, and into production as drift monitors that auto-score a sampled share of live traffic
• The failure flywheel: production failures mined into permanent regression cases, so the same class of error cannot recur silently
• The evaluation evidence pack that services customer model-risk validation (SR 11-7 vendor obligations), written to be read by people outside engineering
• The line's local harness: steering files, specification constitutions, and agent tooling, kept current with platform patterns and pruned when model progress makes a guard unnecessary

• Machine learning or data engineering experience with genuine measurement rigour, and production systems you have measured rather than only built
• Statistics you can defend: cohort design, sample sizing, confidence intervals, and the discipline to say a result is not yet real
• Hands-on evaluation engineering for language models: golden datasets, LLM-as-judge calibration, regression gates in CI, and the failure modes of each
• Strong Python, and comfort operating CI/CD systems
• The willingness to build domain literacy in the seat, working daily with a claims or payments expert
• Daily, hands-on use of AI coding assistants as part of your own workflow

• You have produced evaluation evidence that an external risk, audit, or regulatory function accepted
• Adversarial or red-team dataset construction, and an…