Software Engineer, ML Infra, Autonomy

Rivian is on a mission to keep the world adventurous forever. This goes for the emissions-free Electric Adventure Vehicles we build, and the curious, courageous souls we seek to attract. As a company, we constantly challenge what’s possible, never simply accepting what has always been done. We reframe old problems, seek new solutions and operate comfortably in areas that are unknown.

Our backgrounds are diverse, but our team shares a love of the outdoors and a desire to protect it for future generations.

Rivian Autonomy is building an ML Infrastructure team to give hundreds of ML engineers a training platform they can trust at fleet scale. We are seeking a Staff Software Engineer to help design and build the platform that trains and evaluates our autonomous driving models: the control plane that schedules jobs across accelerator clusters, the storage and I/O layer that feeds them, the observability that tells us where every GPU-hour goes, and the training framework layer that lets ML engineers write model code once and run it on any silicon we operate.

Autonomy at Rivian runs a training fleet of thousands of GPUs on Kubernetes over petabyte-scale sensor data, with additional accelerator types arriving in the coming months. The fleet is fully allocated, so the next step change comes from goodput — how much useful training each GPU-hour delivers. Raising it, through scheduling that keeps large gang jobs fed, a storage and I/O layer that keeps up with the accelerators, and per-workload baselines that make every optimization measurable, is the heart of this role.

The control plane and training fleet already exist and serve every ML engineer in Rivian Autonomy; the scheduling, storage/I/O, observability and framework layers on top of them are largely still to be built. This is a role for someone who wants to set technical direction rather than maintain it. The platform spans four areas — job scheduling and multi-tenant cluster management, training data storage and I/O, observability and workload optimization, and the training framework layer.

You will lead one or two of these areas end-to-end and contribute across the rest; we do not expect one person to be an expert in all four. You will partner closely with the model training teams who are the platform’s customers, with the Cloud Infrastructure team that owns the clusters underneath, and with the Data Infrastructure team that produces the datasets the platform serves.

As an early member of the team, you will help define its technical direction, operating model and future hiring.

• Evolve our control plane into the single entry point for every accelerator cluster we operate, across cloud providers and silicon types: a user asks for N accelerators of a given type, not for a specific cluster.
• Design scheduling that maximizes fleet goodput rather than queue order: gang scheduling for jobs of hundreds of nodes, multi-factor priority and fair sharing across teams, quota borrowing with enforceable reclaim, and execution-time-aware backfill.
• Treat availability as an engineered system: continuous node health checking with automatic cordon, drain and replace, and automatic classification of every failed job (user, out-of-memory, communication, hardware, platform).

• Design the storage tiering between object storage, shared or node-local caches and memory, and the request patterns that keep thousands of concurrent readers from overwhelming the object store.
• Kill the small-file problem for good: shard formats and indexing that serve both fleet-scale shuffled random access and sequential scans, source data stored once and referenced…