More jobs:
Software Engineer - Benchmark Auditor
Job in
San Francisco, San Francisco County, California, 94199, USA
Listed on 2026-09-02
Listing for:
Mercor
Full Time
position Listed on 2026-09-02
Job specializations:
-
Software Development
Software Testing, Python
Job Description & How to Apply Below
Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity — and provide clear, rubric-based written feedback.
Basic Qualifications- 3+ years professional software engineering
- Real open-source contribution or maintainer experience (merged PRs, committer / maintainer roles)
- Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking
- Fluency across common ecosystems (Python and at least one of Java / Go / Type Script / C++)
- Familiarity with SWE-Bench (Verified) or similar repository benchmarks
- Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.)
- Prior code-review or task-grading experience
To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
Search for further Jobs Here:
×