Senior Machine Learning Performance Engineer

XPENG is a leading smart technology company at the forefront of innovation, integrating advanced AI and autonomous driving technologies into its vehicles, including electric vehicles (EVs), electric vertical take-off and landing (eVTOL) aircraft, and robotics. With a strong focus on intelligent mobility, XPENG is dedicated to reshaping the future of transportation through cutting-edge R&D in AI, machine learning, and smart connectivity.

The Mission: We are building the next generation of end-to-end autonomous driving on physical AI foundation models — vision-language-action policies and world models — and the same training stack carries our robotics program. These training runs span thousands of accelerators and days of wall-clock — how fast we train them is how fast we learn. As a Senior Staff Machine Learning Performance Engineer, you will own that speed: working with the modeling teams on their hardest bottlenecks, setting the technical direction for performance, and turning one-off wins into defaults — including the agentic workflows that find the next round of them automatically.

Training Performance at Scale: Own step time, scaling efficiency and long-run stability for foundation-model training across thousands of accelerators — from the data path to kernels, parallelism and communication.

Cross-Team Technical Leadership: Work directly with the modeling, infrastructure, simulation, data-mining and robotics teams on their hardest bottlenecks, and set the performance direction those teams build against.

Reusable by Design: Turn one-off wins into modular components that become defaults across most models and workloads, rather than a patch that helps a single job.

Observability at Scale: Extend performance tooling and metric coverage across the stack, while keeping instrumentation overhead low enough to stay on for production jobs fleet-wide.

Agentic Automation: Build agents that reproduce any job on demand, verify numerical parity and separate a real regression from run-to-run noise — so that most of the optimizations that land are found and fixed by AI rather than by hand.

Master's or PhD in CS, CE or EE, or equivalent practical experience, with 8+ years of industry experience, the majority of it in performance engineering.

Hands-on experience optimizing GPU training or inference workloads in production, at multi-node scale.

A track record of diagnosing performance problems in real production workloads and landing the fixes — often in systems owned by other teams — with the gain measured and verified in production.

In-depth understanding of computer architecture and high-performance computing.

Strong C++ and Python, with a solid command of data structures and algorithms.

Hands-on experience writing and tuning CUDA or Triton kernels.

Fluent in PyTorch, with concrete insight into what makes PyTorch code fast or slow — graph breaks, host-side overhead, synchronization points and memory layout.

Training at the scale of thousands of accelerators, including multi-node failure modes and long-running job stability.

Building profiling, tracing or benchmarking infrastructure that other teams depend on — CUPTI, Perfetto, or equivalent.

Using LLM agents to automate parts of an engineering workflow.

Autonomous driving, robotics, or another domain with hard latency, power and memory constraints.

Publications or open-source contributions in ML systems or performance.

A fun, supportive and engaging environment.

Infrastructures and computational resources to support your work.

Opportunity to work on cutting edge technologies with the top talents in the field.

Opportunity…