Research Engineer, Autonomy VLM

Rivian Staff Research Engineer

Rivian is on a mission to keep the world adventurous forever. This goes for the emissions-free Electric Adventure Vehicles we build, and the curious, courageous souls we seek to attract. As a company, we constantly challenge what's possible, never simply accepting what has always been done. We reframe old problems, seek new solutions and operate comfortably in areas that are unknown.

Our backgrounds are diverse, but our team shares a love of the outdoors and a desire to protect it for future generations.

Role

Summary:

Vision-Language Models (VLMs) are a foundational pillar of our Autonomy stack. In this Staff Research Engineer role, you will play a key role in delivering the overarching VLM strategy, especially training, shipping, optimizing the VLM models, as well as extending to multi-modalities and enabling new use cases, among others. In this role, you will also be responsible to define and deliver VLM-driven solutions to solve some of autonomy's hardest challenges, including automated data mining, handling long-tail distributions, rare edge-case detection, and scene anomaly reasoning.

As part of the model delivery, you will also own the whole end-to-end lifecycle of VLM model delivery: data acquisition, metrics definition, benchmarking, model performance optimization, deployment, feedback loop.

Responsibilities:

• Drive and deliver the VLM model strategy:
  Define, drive and execute the roadmap of VLM model delivery, including training and delivering VLM models, optimization, deployment, as well as the extension to other multi-modalities.
• Accelerate data mining:
  Design and deliver VLM/LLM related models and strategies that power automated data mining, long-tail distributions, rare/edge case detection, and anomaly detection at scale, across multiple modalities (vision, lidar, text, etc).
• Iterate and optimize performance:
  Establish rigorous evaluation and monitoring benchmarks. Identify and root-cause top-tier system anomalies, prioritizing high-impact optimizations to continuously push the needle on performance.
• Cross-functional collaboration:
  
  Partner closely with core Autonomy teams (Perception, Planning, Calibration, Systems, etc) to translate vehicle feature requirements into concrete ML deliverables.
• Influence trade-offs & requirements:
  Define system requirements and guide cross-functional efforts through technical trade-off decisions.

Qualifications:

• Education:
  
  BS, MS, or PhD in Computer Science, Robotics, Electrical Engineering, or a highly related quantitative field.
• Experience:
  
  5+ years of professional experience scaling ML solutions, with a strong focus on the following:
• VLM model training:
  Hands-on experience training or fine-tuning VLMs using modern parameter-efficient techniques (LoRA, QLoRA) and RL alignment.
• Large-scale data mining:
  Proven track record developing VLM/LLM-related techniques for data mining, long-tail distributions, rare cases, safety-critical events.
• Zero/few-shot capabilities:
  Experience with open-vocabulary, zero-shot, or few-shot classification models, particularly in long-tail scenarios.
• System engineering:
  Strong proficiency in Python alongside a solid understanding of modern Perception pipelines, benchmarking tools, and infrastructure.
• Execution:
  Demonstrated ability to root-cause complex issues across a distributed, cross-functional stack in a fast-paced environment.

Preferred Qualifications:

• Experience applying VLMs within the Autonomous Vehicle domain.
• Experience with Auto Prompt Optimization (APO) and automated prompt engineering techniques.
• Experience with spatial grounding in 2D and/or 3D.
• Experience extending foundational models to extra modalities (e.g., LiDAR, Radar, IMU, ego-motion).
• Experience utilizing VLMs or Foundation Models for complex behavior reasoning and planning.
• Experience with onboard edge deployment, cloud inference architectures, and balancing compute/efficiency trade-offs.
• Experience with quantization techniques (PTQ, QAT) and high-performance inference engines like TensorRT.