Research Scientist, Benchmarks Red Team
Job in
San Antonio, Bexar County, Texas, 78201, USA
Listing for:
snorkelai
Full Time
position
Listed on 2026-11-16
Job specializations:
-
Research/Development
Job Description & How to Apply Below
Role overview
Help make evaluations of advanced AI systems more reliable by finding weaknesses in benchmarks and working with their authors to validate and fix them. The work combines hands-on testing of models and agents, research into recurring evaluation failures, and engineering improvements that can be reused across tasks.
Responsibilities- Probe benchmarks for grading errors, ambiguous tasks, exploitable shortcuts, reward hacking, and environment vulnerabilities; reproduce issues so they can be investigated.
- Propose repairs for confirmed problems, implement selected changes with benchmark authors, and check that fixes preserve valid solutions.
- Build or adapt tools and methods that can detect recurring benchmark weaknesses across tasks and datasets.
- Share findings with authors, coordinate improvements, and help prepare updated benchmark releases.
- Contribute to technical reports, open-source tools, publications, and new approaches to evaluating benchmarks.
- Research or engineering experience in machine learning, large language models, AI agents, and evaluation.
- Strong Python skills and experience building, running, and debugging agents or evaluation systems.
- Ability to identify limitations in evaluation methods, develop alternatives, and test them rigorously.
- Experience designing experiments, analyzing model behavior, and investigating unexpected failures.
- Familiarity with agent or language-model benchmarks, evaluation frameworks, or automated graders.
- Ability to take research or engineering work from problem discovery through implementation and validation, and communicate findings clearly.
- Experience with adversarial testing, red teaming, reward hacking, or specification gaming.
- Work on benchmarks for coding agents, computer use, or professional workflows; familiarity with reinforcement-learning environments, agent trajectories, verifiers, or evaluation infrastructure.
- Publications or open-source work in AI evaluation, robustness, or agent systems. A relevant doctorate or equivalent research or industry experience is also valued.
To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction