Verifier Engineer

A reward is only as good as the check behind it. You’ll write the verifiers that decide whether an agent did the work: executable tests, state diffs, rubric graders, and the adversarial cases that catch an agent taking a shortcut.

• Executable checks for coding, data and operational tasks: tests, in variants, state comparisons, output validators.
• Rubric-based graders for tasks without a single right answer, calibrated against expert review.
• Shortcut hunting: ways to get the reward without doing the task, closed before a customer trains on them.
• Verifier precision and recall against held-out human judgements, reported honestly.
• A library of reusable verification primitives across environments.

• Strong software testing instincts and a habit of thinking adversarially about your own code.
• Fluent Python, and comfort reading unfamiliar codebases in other languages.
• Statistical literacy: you can tell when a grader’s agreement rate is good enough and when it isn’t.
• Patience for edge cases. The tenth one matters as much as the first.

• Property-based testing, fuzzing or formal checks.
• Model-as-judge pipelines and their failure modes.