Research Scientist
Job in
Philadelphia, Philadelphia County, Pennsylvania, 19113, USA
Listing for:
Anyone AI Labs
Full Time
position
Listed on 2026-11-05
Job specializations:
-
Research/Development
Job Description & How to Apply Below
- Conduct evaluation research to turn eval targets into original benchmark designs
- Own measurement questions including construct validity, item discrimination, and contamination
- Build evaluation packages with expert-verified ground truth and rigorous QC
- Recruit, calibrate, and review a pool of subject-matter experts across coding, agentic, and STEM fields
- Act as a technical point of contact for AI labs to translate measurement needs into evaluation designs
- Turn research findings into public benchmarks and papers for venues like NeurIPS, ICLR, and ACL
- Research background in ML evaluation or benchmarking
- Deep expertise in LLM/frontier-model benchmarking, specifically code-model and agentic evaluation
- Strong understanding of psychometrics, rubrics, and the measurement problem
- Interest in AI safety, capability elicitation, and robustness
- Ability to manage an expert pool and maintain rigorous standards
- Proficiency in the full research loop from framing to publication
- Fluency in English;
Spanish is a plus
To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction