Head of Research

About the Role

Measuring intelligence is hard, and humans have not been particularly good  proxies we have used — IQ, standardized tests, credentials — have shaped how we develop intelligence and how we value it, often in ways we later regret. AI offers an opportunity to improve this. The field is young, and the methodologies for measuring what these systems can actually do are still being written.

The answers we settle on will shape what gets built, what gets deployed, and which workflows get automated next.

Vals is building the measurement layer for the AI economy: the benchmarks, methodologies, and standards that determine which models ship and where they are trusted. We are hiring a Head of Research to lead this effort.

The hard research questions don’t have textbook answers yet. How do you measure whether an LLM can actually perform a real lawyer’s contract review, a real underwriter’s risk assessment, or a real radiologist’s read? How do you build evaluations that hold up as models improve at gaming them? You will set the direction on how Vals — and by extension, much of the field — answers these questions.

Concretely, you will:

• Advance the science of evaluation. Current methodologies — judge models, human-in-the-loop, static benchmarks — were built for a previous generation of models and break down on long-horizon, real-world tasks. You will develop new paradigms.

• Oversee Vals’ broader research portfolio, setting direction across ongoing projects and new initiatives.

• Publish work that moves the field forward. We want Vals’ research to be cited, not just shipped.

• Recruit and grow a research team alongside the founders.

• Work directly with enterprise customers and lab partners on evaluation problems they actually have.

• A PhD in ML/NLP (in progress or completed), or equivalent industry research track record.

• Deep familiarity with the LLM evaluation landscape: existing benchmarks, their failure modes, judge-model approaches, and human-in-the-loop methodologies.

• A bias toward research that affects what people actually deploy, rather than benchmarks that are easy to game.

• Strong written and verbal communication. You will publish, present, and talk to customers and labs.

• Ability to work in-person, in San Francisco.

• A widely-cited benchmark or eval framework you have built or co-built.

• Prior experience at a frontier lab (Anthropic, OpenAI, Google Deep Mind, Meta FAIR) or a research-led startup.

• Domain depth in one or more of our verticals (legal, finance, insurance, healthcare).

• Experience leading or mentoring other researchers.

• A public research presence: papers, blog posts, talks, or open-source contributions recognized in the field.

• Highly competitive salary and equity. Excellence is well rewarded.

• Relocation and transportation support

• Health/dental insurance coverage

• Lunch and dinner provided, free snacks/coffee/drinks

• 401K plan

• Unlimited PTO

Founding team:
The core methodology behind this platform comes from NLP evaluation research done  raised a $5M seed from top institutional and angel investors. Our team includes members with prior experience at NVIDIA, Meta, Microsoft, Palantir, and HRT. Collectively, we have numerous published citations. Our early team includes Stanford PhDs, ex-Jane Street quants, and the first designer at Snorkel.

Tech stack:
We use Python for most tasks. Our platform is built on Django with a React frontend. All infrastructure is on AWS using CDK for IaC.

What We’re Looking For

• Learning velocity:
  The role involves a wide variety of tasks. We seek someone who can learn new skills and technologies quickly.

• Ownership:
  In a small, talent-dense team, we expect initiative to build where it is needed, with autonomy over consensus. This is especially true for this role.

• Intensity:
  The LLM landscape is rapidly evolving. Foundational labs push the frontier, and the fastest teams win with high-speed execution.

• Solution-oriented mindset:
  Look for opportunities to craft solutions at each step rather than passing hard problems to others.

Further Reading:

• Hugging Face blog on evaluation

• Anthropic’s blog on challenges in evaluation

• New York Times article on issues in benchmarking

• Stanford HAI report showing hallucinations in legal tech tools