AI Benchmark Task Author · Turing Turing · Varies · remote · ago Varies ago
Listing for:
Benture
Full Time, Part Time, Remote/Work from Home
position
Listed on 2026-11-16
Job specializations:
-
Research/Development
Job Description & How to Apply Below
AI Benchmark Task Author | Contractor | Remote | Rate varies | Up to 40 hrs/week with 4 hrs PST overlap
Turing, based in San Francisco, works with frontier AI labs and enterprises on AI research, data, and training. It is hiring an analytical researcher and red-teaming specialist to build evidence-based web-research benchmark tasks that expose weaknesses in state-of-the-art AI models (e.g., Claude Opus 5 Max). You will discover hard-to-surface facts, build precise research tasks around them, test them directly against leading LLMs, and diagnose exactly how the model's search and reasoning paths fail.
Whatthe Work Involves
- Research niche topics to find factual ground truths that cannot be surfaced with simple, direct search queries.
- Verify facts against primary or highly credible sources, cross-check for contradictory evidence, and document exact source URLs.
- Draft clear, answerable benchmark tasks designed to expose structural weaknesses in model research strategies.
- Run tasks against designated SOTA models to confirm they reliably trigger the intended model failures.
- Trace model run logs, search trajectories, and intermediate outputs to pinpoint where and why the model failed, distinguishing genuine model limitations from ambiguous prompts.
- Apply validated "model-breaking" research patterns across new domains; strong performers also review tasks authored by peer trainers.
- 1+ years in investigative research, fact-checking, AI benchmark task authoring, red-teaming, or technical QA review.
- Experience authoring, testing, or reviewing tasks for AI benchmarks such as GAIA, Browse Comp, SimpleQA, SWE-bench, Terminal-Bench, or Humanity's Last Exam (HLE).
- Intermediate-to-advanced web search and OSINT skills; ability to navigate academic archives, local records, or obscure databases.
- Ability to inspect model reasoning chains, search logs, and source selection to diagnose root causes of failure.
- Precision writing: clear, unambiguous benchmark prompts with bulletproof, primary-sourced answer keys.
- Bachelor's degree or higher in a relevant engineering domain (described as a must in the listing).
- Experience designing prompt-injection or reasoning-trap tasks that break SOTA models.
- Advanced OSINT capabilities or published investigative/academic research demonstrating deep source verification.
- Niche domain expertise in areas rich with hard-to-find records (e.g., local government filings, regional sports history, legal archives, historical patents).
- Experience as a lead reviewer, benchmark auditor, or quality owner on AI evaluation projects.
- Up to 40 hours per week, with 4 hours of overlap with PST.
- Contractor engagement, up to 4 weeks long.
- Hourly rate is not stated in the listing.
To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction