Senior Principal AI Engineer

A Moving Experience.

What You Will Work On

• Design andoperatedistributed training systems for large neural networks(autoregressive, diffusion,State Space Modelsetc.)across GPU clusters

• Optimisemulti‑node, multi‑GPU execution tomaximizethroughput and utilization

• Diagnose&resolve bottlenecks across compute, memory, and network

• Improve training stability and fault tolerance at scale

• Partner with research and applied ML teams to productionizelarge‑model training pipelines

Core Responsibilities

• Distributed Training Infrastructure

• Build andoptimizeGPU cluster orchestration using:

• Slurm

• Kubernetes

• Ray

• RunAI

• Ensure efficient scheduling, isolation, and fairness across training workloads

• Communication & Networking

• Optimize and debug distributed communication using:

• NCCL

• RDMA

• Infini Band

• NVLink

• Minimizenetworking bottlenecks that dominateend‑to‑endtraining time

• Training Frameworks

• Scale large-model training using:

• Py Torch Distributed

• Megatron‑LM

• Deep Speed

• Ownmulti‑nodelaunch configurations, failure recovery, and performance tuning

• Memory & Performance Optimization

• Apply advanced memory optimization techniques:

• Activation checkpointing

• ZeRO(Stage 1–3) and offload strategies

• Balance compute, memory, and communication to push model size and batch scale

What Success Looks Like

• GPUutilizationconsistently stays high (>80–90%)

• Training scales cleanly from single node to dozens or hundreds of GPUs

• Communication overhead is minimized and predictable

• Large training jobs run stably for days or weeks without failure

• New models can be trained faster, larger, and more reliably than before

Required Experience & Skills

• Strongly Required

• Deephands‑on experience with distributed systems or ML systems

• Experience runninglarge‑scale workloads on GPU clusters

• Production experience with
  
  PyTorchdistributed training

• Strong understanding of parallelism strategies (data, tensor, pipeline parallelism)

• Low‑level understanding of GPU communication and networking

• Critical Technical Skills

• GPU orchestration:
  
  Slurm, Kubernetes, Ray,RunAI

• Communication libraries: NCCL, RDMA, Infini Band,NVLink

• Training frameworks:
  
  Py Torch Distributed ,Megatron‑LM,Deep Speed

• Memoryoptimisation: activation checkpointing,ZeROoffload techniques

Common Problems You’llBe Solving

• Many teams fail at scale because:

• GPUutilizationis low despite large clusters

• Networking and communication dominate training time

• Training jobs crash or become unstable at large scale

• You will be explicitly focused oneliminatingthese failure modes.

Ideal Background

• This role is a strong fit for individuals who have worked as:

• ML Systems Engineer

• Distributed Systems Engineer

• AI Infrastructure Engineer

• HPC Engineer transitioning into ML

• Experience working with large language models or foundation models is a strong plus, but deep systemsexpertiseis valued over pure model architecture experience.

Why This Role Matters

Without robust distributed training infrastructure, progress on largemodelsstalls. This role directly enables:

• Larger models

• Faster iteration cycles

• More reliable research-to-production pipelines

You will be building the foundation that makeslarge‑scaleAI possible.

Cerence Inc. (Nasdaq: CRNC and ) is the global industry leader in creating unique, moving experiences for the automotive world. Spun out from Nuance in October 2019, Cerence is a new, independent company that has quickly gained traction as a leader in the automotive voice assistant space, working with all of the world’s leading automakers – from Ford and Fiat Chrysler to Daimler, Audi and BMW to Geely and SAIC – to transform how a car feels, responds and learns.

Its track record is built on more than 20 years of industry experience and leadership and more than 500 million cars on the road today across more than 70 languages.

AsCerencelooks to the future and continues an…