Machine Learning Engineer, LLM Inference Optimization

About Nebius:

Nebius is leading a new era in cloud infrastructure for the global AI economy. We are building a full-stack AI cloud platform that supports developers and enterprises from data and model training through to production deployment, without the cost and complexity of building large in-house AI/ML infrastructure.

Built by engineers, for engineers. From large-scale GPU orchestration to inference optimization, we own the hard problems across compute, storage, networking and applied AI.

Listed on Nasdaq (NBIS) and headquartered in Amsterdam, we have a global footprint with R&D hubs across Europe, the UK, North America and Israel. Our team of 1,500+ includes hundreds of engineers with deep expertise across hardware, software and AI R&D.

Nebius Token Factory is building production-grade inference for open-source, customer, and internally optimized models. This role owns the path from model artifact to fast, reliable, cost-efficient endpoint. The work sits at the intersection of model internals, inference engines, serving architecture, benchmarking, and production deployment.

• Own a platform-level optimization pillar such as quantization, speculative decoding, KV-cache optimization, engine selection, or long-context serving.

• Create reusable evaluation, benchmarking, and rollout frameworks across multiple model families.

• Provide technical direction to MLEs, Applied Scientists, GPU kernel engineers, and infrastructure engineers on shared performance goals.

• Optimize LLM and VLM endpoints for latency, throughput, memory efficiency, GPU utilization, quality, and cost per token.

• Deploy, configure, benchmark, and extend inference engines such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, NVIDIA Dynamo, or similar systems.

• Build and product ionize model-compression workflows, including quantization, quantization-aware training, distillation, low-bit serving, and accuracy recovery.

• Implement or integrate speculative decoding, draft-model approaches, KV-cache optimization, prefix caching, chunked prefill, continuous batching, and disaggregated prefill/decode serving.

• Build reproducible benchmark harnesses for TTFT, TPOT, tokens per second per GPU, p95/p99 latency, GPU memory, reliability, and cost per token.

• Partner with GPU kernel engineers and platform engineers to diagnose bottlenecks across model code, kernels, runtime, scheduler, gateway, and cluster layers.

• Write clear design docs, performance reports, rollout plans, and customer-facing technical explanations.

• Strong Python and PyTorch engineering skills.

• Hands-on experience deploying or optimizing LLM, VLM, or high-throughput transformer inference systems.

• Practical knowledge of at least one modern inference stack such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, NVIDIA Dynamo, Ray Serve, KServe, or equivalent internal systems.

• Strong understanding of transformer inference bottlenecks, including KV cache, attention, memory bandwidth, batching, parallelism, and long-context serving.

• Ability to reason quantitatively about latency, throughput, quality, utilization, and cost tradeoffs.

• Strong communication skills and ability to collaborate with research, kernel, infrastructure, product, and customer teams.

• Experience with quantization-aware training, post-training quantization, FP8, INT8, INT4, NVFP4, MXFP4, AWQ, GPTQ, Smooth Quant, or related techniques.

• Experience with distillation, speculative decoding, EAGLE, Medusa, multi-token prediction, or other inference acceleration methods.

• Experience with agentic workloads, including tool calling, structured outputs, streaming APIs, high concurrency, and multi-step orchestration.

• CUDA or Triton familiarity, even if the role is not primarily a kernel-engineering role.

• Open-source contributions to vLLM, SGLang, TensorRT-LLM, Flash Infer, LMCache, PyTorch, Triton, Ray, KServe, or related projects.

Benefits & Perks:

• Competitive compensation
• Career growth and learning opportunities
• Flexibility and ownership
• Collaborative and innovative culture
• Opportunity to work on impactful AI projects
• International environment and talented teams

What's it like to work at Nebius:

Fast moving
- Bold thinking
- Constant growth
- Meaningful impact
- Trust and real ownership
- Opportunity to shape the future of AI

Equal Opportunity Statement:

Nebius is an equal opportunity employer. We are committed to fostering an inclusive and diverse workplace and to providing equal employment…