Head of Evaluations Research

Location: New York

# Head of Evaluations Research Technical Staff

Research NYC $450K – $750K
• Offers Equity##
** About Aaru
** Aaru builds simulations of human behavior. Each simulation contains a population of agents, each representing a person who could plausibly exist in the real world and capable of making decisions within a modeled environment. Companies and institutions use these simulations to test consequential choices before committing, from product launches and policy changes to critical communications. Because the agents are simulated rather than recruited, they can reason through complex hypotheticals without fatigue or the response effects common in human studies.##

** The role
** Evaluation Research determines whether Aaru's populations, predictions, and simulations correspond closely enough to the real world to support consequential decisions. The function defines what should be measured, develops the methods for measuring it, and produces the evidence Aaru uses to improve its systems and describe their capabilities.

As Head of Evaluation Research, you will set Aaru's evaluation charter and lead the research needed to test its core systems. The strongest evaluations will be grounded in observed behavior and outcomes, including transactions, product usage, operational records, resolved events, and longitudinal decisions. You will work across Population Research, Prediction Research, Simulation Engineering, and customer-facing teams while preserving the independence needed to identify inconvenient results.

This is a hands-on research leadership role. You will design studies, construct evaluation datasets, write analysis code, inspect individual failures, and develop new measurements when existing benchmarks are inadequate. You will also recruit and lead a small team that can combine scientific rigor with a practical understanding of how research and engineering systems improve.##
** What you will do
*** Define a coherent evaluation agenda across population construction, predictive systems, agent behavior, group dynamics, and end-to-end simulations.
* Turn broad questions about realism, accuracy, and usefulness into measurable constructs, decisive experiments, and clear decision criteria.
* Build tests of population quality that assess whether each generated person forms a coherent whole, whether the population reproduces important relationships in the data, and whether rare but plausible profiles are represented.
* Evaluate forecasts and other predictive outputs using temporal holdouts, prospective outcomes, calibration, ranking quality, subgroup performance, and the real cost of different errors.
* Compare simulations with transactions, behavioral traces, product adoption, operational outcomes, market movements, and other records of what people actually did.
* Design longitudinal and interaction-based evaluations that test how agents change over time, respond to new information, and influence one another.
* Develop end-to-end studies that show whether better components lead to better answers for the decisions customers use Aaru to make.
* Find failures hidden by aggregate metrics, especially those concentrated in important subgroups, rare cases, or changing environments.
* Establish strong baselines, clean holdouts, contamination controls, and statistical standards appropriate to each research question.
* Create diagnostic evaluations that help researchers identify why a system failed and whether a proposed fix generalizes.
* Work with Simulation Engineering to make evaluations repeatable and versioned, and convert production outcomes and customer failures into durable test cases.
* Produce evidence for customers and the public that is reproducible, appropriately scoped, and clear about uncertainty and limitations.
* Communicate negative and inconclusive results with the same care as positive findings.
* Hire, mentor, and lead exceptional evaluation researchers and research engineers while remaining a direct contributor.##
** Representative research directions
*** Construct a population using data available at one point in time, then test whether its future transactions, choices, or behavioral outcomes match what…