Skip to content
×

SWE-Bench Task Auditor

Job in Chicago, Cook County, Illinois, 60601, USA
Listing for: Parailabs
Full Time position Listed on 2026-11-19
Job specializations:
Salary/Wage Range or Industry Benchmark: 120000 - 180000 USD Yearly USD 120000.00 180000.00 YEAR

Job Description & How to Apply Below

Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity — and provide clear, rubric-based written feedback.

Basic Qualifications
  • 3+ years professional software engineering
  • Real open-source contribution or maintainer experience (merged PRs, committer / maintainer roles)
  • Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking
  • Fluency across common ecosystems (Python and at least one of Java / Go / Type Script / C++)
Preferred Qualifications
  • Familiarity with SWE-Bench (Verified) or similar repository benchmarks
  • Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.)
  • Prior code-review or task-grading experience

We are looking to fill the SWE-Bench Task Auditor position at Parailabs in United States.

To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction

Search for further Jobs Here:

SEARCH JOBS: 10+ Million Listings
(Try combinations for better Results! Or enter less keywords for broader Results)
Education Level
Experience Level (years)
Posted in last:
Salary