Skip to content
×

Research Scientist, Benchmarks Red Team

Job in San Antonio, Bexar County, Texas, 78201, USA
Listing for: snorkelai
Full Time position Listed on 2026-11-16
Job specializations:
Salary/Wage Range or Industry Benchmark: 120000 - 180000 USD Yearly USD 120000.00 180000.00 YEAR

Job Description & How to Apply Below

Position: Research Scientist, Open Benchmarks Red Team
Role overview

Help make evaluations of advanced AI systems more reliable by finding weaknesses in benchmarks and working with their authors to validate and fix them. The work combines hands-on testing of models and agents, research into recurring evaluation failures, and engineering improvements that can be reused across tasks.

Responsibilities
  • Probe benchmarks for grading errors, ambiguous tasks, exploitable shortcuts, reward hacking, and environment vulnerabilities; reproduce issues so they can be investigated.
  • Propose repairs for confirmed problems, implement selected changes with benchmark authors, and check that fixes preserve valid solutions.
  • Build or adapt tools and methods that can detect recurring benchmark weaknesses across tasks and datasets.
  • Share findings with authors, coordinate improvements, and help prepare updated benchmark releases.
  • Contribute to technical reports, open-source tools, publications, and new approaches to evaluating benchmarks.
Requirements
  • Research or engineering experience in machine learning, large language models, AI agents, and evaluation.
  • Strong Python skills and experience building, running, and debugging agents or evaluation systems.
  • Ability to identify limitations in evaluation methods, develop alternatives, and test them rigorously.
  • Experience designing experiments, analyzing model behavior, and investigating unexpected failures.
  • Familiarity with agent or language-model benchmarks, evaluation frameworks, or automated graders.
  • Ability to take research or engineering work from problem discovery through implementation and validation, and communicate findings clearly.
Nice to have
  • Experience with adversarial testing, red teaming, reward hacking, or specification gaming.
  • Work on benchmarks for coding agents, computer use, or professional workflows; familiarity with reinforcement-learning environments, agent trajectories, verifiers, or evaluation infrastructure.
  • Publications or open-source work in AI evaluation, robustness, or agent systems. A relevant doctorate or equivalent research or industry experience is also valued.
To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction

Search for further Jobs Here:

SEARCH JOBS: 10+ Million Listings
(Try combinations for better Results! Or enter less keywords for broader Results)
Education Level
Experience Level (years)
Posted in last:
Salary