Safety and Alignment

Fleet studies how environments produce safe intelligence. We believe safety and economic value increasingly reinforce each other: as agents take on more consequential work, their value depends on whether they can be trusted with that responsibility.

As reinforcement learning takes a larger role in post-training, the design of tasks, environments, and feedback becomes increasingly consequential for downstream alignment. RL can reinforce sound judgment, but it can also amplify unintended behaviors. We believe studying and scaling the data and training signals that produce more aligned agents is one of the most impactful ways to contribute to AI safety.

Backed by Sequoia Capital, Menlo Ventures, BCV, and SV Angel.

We’re looking for an Applied Research Scientist to work on robust reward design, implement comprehensive telemetry, and design production systems to help ensure Fleet's RL tasks make capable agents more aligned. You’ll work with data and model trajectories to help develop oversight systems to detect misaligned behaviors. You’ll also work directly with safety researchers at frontier labs to help co-design new RL environments and tasks for downstream alignment training and evaluation.

• Reward Design and Telemetry at Production Scale. Study how task and verifier design affect model-alignment incentives, then improve both.
  • Off-target behaviors from task assumptions and incentives. Investigate how unstated behavioral constraints, unsolvable tasks, and rewards that ignore how success is achieved affect how models behave during evaluation, across many different domains.
  • Telemetry and Verifier Generation Pipelines. Use the insights learned from the above to build
    1) the telemetry needed for customers to have complete observability over undesirable model behaviors and
    2) the scalable pipelines needed for producing verifiers that translate recorded measurements into qualitative behavioral categories.
  • Production Oversight Systems. Build out rigorous trace-based analysis systems that measure agent misbehavior across all internal Fleet evaluations.
• Safety evaluations and benchmark analysis. Develop rigorous safety measurements and assess the risks in existing Fleet tasks and external benchmarks.
  • Safety benchmarks. Build held-out and adversarial evaluations to determine the safety profile of existing tasks. Audit Fleet tasks, data, and graders for contamination, poisoning, ambiguity, and shortcuts.
  • External benchmark analysis. Examine widely used benchmarks, datasets, and task designs for incentives that could reinforce unsafe behavior.
• Specialization in high-impact domains. Get involved in domain-specific safety research with frontier labs, including cybersecurity. Work with experts to identify consequential failure modes and design curricula, tasks, and evaluations aimed at preventing incidents like the Hugging Face breach.

• A willingness to work directly with the data: inspect examples and trajectories, question labels and graders, investigate anomalies, and trace a surprising result to its cause.
• Strong practical engineering skills—you should be able to build the systems needed to test and scale your ideas.
• A record of strong empirical experience in alignment, model behavior, model evals, or a closely related field.

Prior research experience via safety fellowships (e.g. MATS, Astra), a safety organization (e.g. METR, Redwood, Epoch AI), or academia is helpful but not necessary.

San Francisco (on-site). Highly competitive salary + equity.