Principal Applied Scientist
-
IT/Tech
Job Description & How to Apply Below
This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for a Principal Applied Scientist based in the United States.
We are seeking an experienced Principal Applied Scientist to lead the development of a rigorous evaluation framework for enterprise AI agents. In this research-intensive, customer-facing role, you will work closely with engineering teams to assess planner models, execution models, and inference optimization strategies through controlled, statistically sound experiments. You will own both the experimental methodology and the production-quality tools that make evaluation repeatable, from realistic user simulation and isolated experiment execution to statistical analysis and automated insights.
Your work will help distinguish meaningful improvements from inconclusive results, quantify the relationship between quality and cost, and translate technical findings into measurable business value. You will also provide technical leadership, establish evaluation standards, and deliver actionable recommendations that support enterprise deployment decisions.
- Design and lead an end-to-end evaluation framework for planner-executor agents, covering scenario definition, user simulation, configuration, execution, grading, comparison, and insight generation.
- Define scenario schemas, grading specifications, and validation processes in collaboration with engineering teams, ensuring scenarios capture goals, personas, starting states, and expected outcomes.
- Develop an LLM-powered user simulator capable of realistic multi-turn conversations, using hidden goals and stop conditions, and validate its behavior against production interaction logs.
- Build isolated, parallel experiment execution systems with configurable runs and adapters that route model traffic through existing LLM gateways.
- Develop and implement an initial selection of optimization plug-ins, such as failure-based model cascading, cache-aware model switching, and prompt caching.
- Instrument model calls and integrate tracing systems to attribute token consumption, cache utilization, latency, and costs to individual requests.
- Establish grading methods based primarily on verified end states, with optional LLM-based evaluation of answer and planning quality.
- Calculate and report key performance indicators, including verified success rates, cost per successful task, p95 latency, and failures by pipeline stage.
- Apply rigorous statistical methods, including paired comparisons, confidence intervals, win/loss/tie analysis, and non-inferiority testing, while clearly identifying inconclusive findings.
- Translate experimental results into projected financial impact at production scale and develop cost-versus-quality analyses to guide optimization decisions.
- Build an automated insight engine that uses statistical computation for all reported metrics and LLM-generated explanations to communicate evidence-backed findings.
- Develop a locally hosted, API-first dashboard with launch, live monitoring, results, comparison, insights, and tracing capabilities, supported by straightforward deployment and integration documentation.
- Lead staged experimentation campaigns covering production baselines, model screening, optimization testing, and validation of the strongest combinations.
- Assess how well agent performance generalizes across scenario categories, potentially incorporating data analytics benchmark scenarios.
- Deliver final recommendations with confidence intervals and projected dollar impact, alongside technical reports, runbooks, and integration guides.
- Collaborate with engineers and business leaders to establish evaluation criteria, success thresholds, and acceptable quality…
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction