Senior Specialist​/Manager, AI Safety Data and Evaluation Program

Cambridge, MA USA;
London, UK;
San Francisco, CA USA

We're building a talent-dense, high-agency AI safety team at Lila that will engage all core teams within the organization (science, model training, lab integration, etc.), to identify and mitigate risks from scientific superintelligence while retaining the core scientific discovery capabilities of the stack. The initial focus of this team will be to build and implement a bespoke safety strategy for Lila, tailored to its specific goals and deployment strategies.

This will involve technical safety strategy development, broader ecosystem engagement, safety-focused evaluations, safety systems to mitigate risks, and a research agenda that explores longer-term needs such as oversight of superintelligent scientific systems.

High-quality safety depends on the right expertise, consistent and repeatable workflows and high levels of interpretability. At Lila Sciences, you will turn scientific risk questions into well-run expert-data and evaluation programs to build this evidence-base.

You will bring a strong working understanding of AI safety and scientific misuse risks to connect internal risk mapping and threat modeling with expertise across Lila and external specialists. Your primary responsibilities will be program execution and data quality: organizing contributors, running red-teaming and evaluation campaigns, and ensuring their outputs meet agreed standards. You will also be responsible for gathering targeted high-impact feedback from partners in the AI Safety community on our different aspects of our approach to Safety.

You will work closely with scientists who frame the research questions and interpret their significance, while creating the operational conditions for their expertise to have an impact. Ultimately, you will play a key role in generating the evidence that informs safety decisions made by senior leadership at Lila.

What You'll Be Building

• Translate agreed scientific briefs into executable plans covering tasks, contributor needs, instructions, milestones, quality standards and delivery expectations.
• Source and coordinate external experts with scientific colleagues, and organize access to relevant expertise within Lila.
• Manage contributor onboarding, task assignment, training and support so specialists can contribute effectively and consistently.
• Run expert-data collection, annotation, red-teaming and evaluation programs, tracking progress and resolving delivery problems.
• Establish quality controls covering completeness, consistency, provenance and adherence to rubrics; coordinate calibration, review, disagreement resolution and rework with scientific owners.
• Maintain clear records, versioned guidance and usable datasets, and improve workflows based on quality findings and researcher feedback.
• Identify questions that require scientific judgment and bring them to the appropriate specialists, ensuring decisions are reflected in program materials and subsequent work.

What You'll Need to Succeed

• Experience delivering expert-data, research-operations, evaluation or similarly demanding programs involving specialist contributors.
• A substantial working understanding of AI safety, including how threat models, capability evaluations, red teaming and safeguards inform one another. You should be able to interpret a safety research brief, recognize its assumptions and the consequences of data-quality failures, and identify when a question needs scientific escalation.
• Evidence of applying this safety understanding in research, evaluation or expert-data work, including how it shaped the scoping, execution or review of a program.
• Strong operational judgment and the ability to manage…