Training Performance Engineer - Acceleration

About the role

Kaiko trains its own foundation models for clinical work. The program runs on open-weight MoE bases in the hundreds‑of‑billions to trillion‑parameter range.

You own throughput on our Blackwell training cluster — instrument runs, identify utilization gaps, and ship optimizations that push MFU, wall‑clock, and uptime. You work alongside research as new architectures and phases land on the cluster.

The hard problems are low‑precision training, modern attention variants on open‑weight MoE bases at the kernel level, and MoE parallelism tuned to the cluster fabric.

You will be based in either The Netherlands or Switzerland, with the expectation of spending at least 50% of your time at the office.

• Instrument and analyze runs — MFU, throughput, uptime — and close gaps against predicted wall‑clocks.
• Benchmark NCCL collectives over Infini Band and NVLink — including rail/topology behaviour and congestion at scale, and keep a current picture of what the fabric delivers.
• Drive low‑precision training in our stack and validate the speed‑up.
• Tune MoE parallelism (TP / PP / CP / EP / DP) per phase and characterise expert‑parallel comm cost on the cluster fabric.
• Land custom attention‑variant kernels (e.g. hybrid, latent‑attention) into the training stack.

• Deep GPU systems experience, with kernel‑level CUDA / Triton work and comfort with CUTLASS, Flash Attention, Pytorch and Nsight profiling.
• Production experience with NCCL on Infini Band or equivalent high‑bandwidth interconnects.
• Parallelism literacy: TP / PP / CP / EP / DP under memory, comm, and MFU constraints.
• Tracks the relevant systems literature and brings it into the stack.

• Low‑precision training (FP8, expert‑only quant, dynamic loss scaling).
• Sparse / hybrid / MLA attention at the kernel level.
• Has shipped large‑scale MoE training in production — pre‑training, SFT, or RL.
• Stack experience with Megatron, NeMo, or comparable.

• An attractive and competitive salary, a good pension plan, and 25 vacation days per year.
• Great offsites and team events to strengthen the team and celebrate successes together.
• A EUR 1000 learning and development budget to help you grow.
• Autonomy to do your work the way that works best for you, whether you have a kid or prefer early mornings.
• An annual commuting subsidy.