Skip to content
×

Member of Technical Staff, Evaluations

Listing for: Bake AI Inc.
Full Time position Listed on 2026-11-05
Job specializations:
Salary/Wage Range or Industry Benchmark: 180000 - 240000 USD Yearly USD 180000.00 240000.00 YEAR

Job Description & How to Apply Below

Bake AI builds environments, evaluations, and sandbox infrastructure for self-evolving AI. We are hiring one Member of Technical Staff to develop evaluations grounded in real-world tasks and workflows. This is a full-time position based in San Mateo, California.

You will turn open-ended questions about model capability and reliability into evaluations that teams can trust. The work starts with understanding what successful performance means in practice: completing a task, using tools correctly, handling incomplete information, and producing an outcome that meets the user's needs. You will work with researchers, engineers, and domain experts to translate those requirements into representative tasks and defensible measures of success.

You will own the path from task design and verification through execution, analysis, and ongoing use. That includes building the evaluation software, inspecting model behavior, and investigating whether a score reflects a meaningful improvement. We are looking for someone who combines strong engineering with careful experimental judgment and can carry an ambiguous problem through to a useful, reproducible result.

What you’ll do
  • Design evaluations for realistic work, including tasks that require multiple steps, external tools, and interaction with changing state. Define success criteria that reflect the quality and completeness of the final outcome.
  • Build task environments, datasets, and reference solutions with researchers and domain experts. Establish appropriate difficulty, coverage, and held-out cases, and document what an evaluation can and cannot tell us.
  • Develop executable verifiers and grading systems. Where human or model judgment is needed, create clear rubrics, calibrate against expert review, and investigate disagreement, bias, and scoring shortcuts.
  • Build reproducible evaluation pipelines on sandbox infrastructure, with controlled configurations, explicit resource budgets, and traceable inputs, tool calls, outputs, and artifacts.
  • Analyze model and agent trajectories to identify failure modes in planning, tool use, reasoning, and recovery. Distinguish model limitations from ambiguous tasks, environment failures, and errors in the evaluator.
  • Design controlled comparisons and repeated trials, quantify uncertainty, and examine performance across task types. Evaluate reliability, quality, latency, and cost alongside aggregate scores.
  • Turn observed failures into targeted evaluations and regression suites. Audit for data leakage, reward hacking, and brittle scoring so that improvements remain meaningful as models and workflows change.
  • Work with partner teams to make evaluation findings actionable, communicate limitations clearly, and maintain the tools and datasets used to track progress over time.
What you’ll bring
  • Evidence of independently building a substantial evaluation, benchmark, research tool, or ML system. You can explain the question it addressed, how you established validity, and the decisions its results supported.
  • Strong Python skills and practical software engineering experience. You can build reliable execution and data-processing pipelines, integrate model and tool APIs, and debug unfamiliar systems.
  • Hands-on experience evaluating language models or agents, including task design, grading, error analysis, and interpretation of results beyond a single headline metric.
  • Solid experimental and statistical judgment: representative sampling, controlled comparisons, repeated measurements, uncertainty, and awareness of confounding factors.
  • The ability to turn an ambiguous workflow into concrete tasks and testable outcomes, while recognizing where automated checks are insufficient and expert judgment is necessary.
  • Careful technical writing and the ability to…
To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction

Search for further Jobs Here:

SEARCH JOBS: 10+ Million Listings
(Try combinations for better Results! Or enter less keywords for broader Results)
Education Level
Experience Level (years)
Posted in last:
Salary