Member of Technical Staff - Research

Directly builds and evaluates LLM benchmarks; mentions a  vibe-coding  benchmark in media, so closely related to vibe-coding evaluation and measurement.

Design and build next-generation benchmarks to evaluate real-world capabilities of large language models (LLMs). Lead research and engineering efforts to create rigorous, publishable evaluations and work with model labs and enterprises to shape how foundation models are measured.

Member of Technical Staff - Research responsible for designing and implementing novel, high-impact benchmarks that assess challenging real-world capabilities of large language models (LLMs). The role focuses on rigorous research, benchmark validity, and collaboration with foundation model labs and enterprise partners.

• Design and develop novel benchmarks that evaluate real-world LLM capabilities
• Conduct research to ensure benchmarks are valid, reliable, and meaningful
• Analyze model performance across benchmarks and articulate results
• Collaborate with foundation model labs, enterprises, and infrastructure teams to implement benchmarks at scale
• Publish research findings and contribute to the evaluation research community
• Stay current with developments in LLM capabilities and evaluation methodologies

• Advanced research experience (Master’s or PhD in Computer Science, NLP, Machine Learning, or related field; exceptional undergraduates may be considered)
• Publication track record in reputable venues (e.g., NeurIPS, ICML, ACL, EMNLP)
• Strong understanding of experimental design, statistical analysis, and evaluation frameworks
• Proficiency in Python for research and experimentation
• Strong communication skills for both technical and non-technical audiences
• Experience collaborating in research teams and integrating feedback
• Demonstrated track record of impactful research work / portfolio

• Prior experience in LLM evaluation or benchmarking research
• Familiarity with foundation model architectures and capabilities
• Experience working with industry partners or in applied research settings
• Backgrounds in human-computer interaction, psychology, or domain-specific evaluation
• Experience at early-stage startups or research labs
• Contributions to open-source evaluation tools or datasets
• Python
• Django
• React (frontend)
• AWS (infrastructure)
• AWS CDK (Infrastructure as Code)

• In-person team based in San Francisco. Relocation and transportation support offered.

• Compensation range: $140,000 - $185,000 (USD)

• Highly competitive salary and meaningful ownership
• Relocation and transportation support
• Health and dental insurance coverage
• Lunch and dinner provided; free snacks, coffee, and drinks
• Unlimited PTO
• $1,500 housing stipend (within one-mile radius)
• Opportunities to collaborate with leading AI labs and publish research findings

Python Django React AWS CDK

Research Benchmark Design Experimental Design Statistical Analysis Data Analysis Communication Collaboration Publication / Academic Writing Project Ownership Rapid Learning Problem Solving

Senior

USD 140,/year

Full-time

• Relocation and transportation support
• Lunch and dinner provided
• Free snacks/coffee/drinks
• $1,500 housing stipend (within one-mile radius)
• Highly competitive salary
• Opportunity to publish and present work