Software Benchmark Auditor: Reproducibility & Quality
Listed on 2026-10-11
-
Quality Assurance - QA/QC
AI QA / Validation Engineer
Obsidian in San Francisco seeks a reviewer and assessor of benchmark tasks for frontier AI labs. You will evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate models, and review repository-level tasks, reference patches, and test harnesses.
You will provide rubric-based written feedback, detect leakage or reward hacking, and help improve grading integrity across the benchmark suite.
We are seeking a motivated Software Benchmark Auditor:
Reproducibility & Quality to join Mercor in San Francisco, CA, United States.
Learn more about the Software Benchmark Auditor:
Reproducibility & Quality role in the description above.
We appreciate your interest in this position.
Join Mercor and contribute to our ongoing work.
Take a moment to read everything above and see whether this role is right for you.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).