Senior Software Engineer, ML Evaluation & Introspection Infrastructure

Aurora’s mission is to deliver the benefits of self-driving technology safely, quickly, and broadly.

The Aurora Driver will create a new era in mobility and logistics, one that will bring a safer, more efficient, and more accessible future to everyone.

At Aurora, you will tackle massively complex problems alongside other passionate, intelligent individuals, growing as an expert while expanding your knowledge. For the latest news from Aurora, visit aurora.tech or follow us on Linked In.

We’re searching for a Senior Software Engineer, ML Evaluation & Introspection Infrastructure to join our Behavior Analysis team, reporting to the team’s engineering manager. Aurora’s trucks already haul freight on Texas highways with no one on board, and every new version of the learned model that drives them has to prove, on real driving data, that it is better and safer than the one before.

Behavior Analysis builds the systems that make that call: the evals that compare model versions, the embedding search that makes every driving scenario findable, and the introspection tools that show why a model did what it did. Engineers build more than 100 training and eval datasets a week through our tools. We’re a small team, so each engineer owns whole systems, and we work agent-first: our tools are built for AI agents to drive as easily as engineers do.

You’ll own core pieces of that loop and help lead what comes next: putting LLMs, VLMs and agents to work on analysis, so questions that take days to answer today take minutes.

• Build and scale evaluation pipelines that compare model versions on large driving datasets: offline evals, slice-based metrics, regression detection, and model-vs-model reports.
• Own embedding and vector-search pipelines over driving data, versioned per model, so engineers can search, cluster and compare how any model represents the world. This is where you’d start: owning embedding generation and search end to end.
• Build model introspection infrastructure (attention, attribution, ablations) that turns “why did the model do that?” from a research project into a query.
• Help define how frontier LLMs, VLMs and agents change ML analysis here: triaging, labeling, clustering and summarizing driving scenarios and eval results. We already have a prototype using LLMs for data curation, and you’d help decide where it goes next.
• Work side by side with the ML engineers training these models to decide what to measure, then make those measurements trustworthy, observable and cheap enough to run on every candidate model.
• Build the web app and CLI that engineers and AI agents use to find datasets, training runs, evals and their costs, designed with agents as first‑class users.

• Typically 5+ years of professional software engineering experience building backend, data or ML infrastructure.
• Strong Python, and proficiency with SQL.
• Designed and operated large-scale batch or data pipelines on cloud infrastructure, including backfills, retries and cost management.
• Built systems that measure or evaluate something: metrics, experimentation, evaluation or analytics platforms.
• Experience with distributed compute and container orchestration (e.g. Kubernetes, AWS Batch or similar).

• ML evaluation infrastructure: eval harnesses, golden datasets, regression detection, A/B model comparison.
• Embeddings and vector search, or model introspection and interpretability tooling.
• Using LLMs, VLMs or agentic workflows to automate analysis, labeling or data curation.
• Autonomous vehicles, robotics or other large-scale sensor data.
• Observability for data systems (metrics, alerting, SLOs), and everyday use of AI coding agents.
• Coming from backend or…