Senior Principal Software Engineer, LLM Inference & GPU Optimization

Senior Principal Software Engineer, LLM Inference & GPU Optimization

Remote, United States

$230,000 to $280,000 base + bonus + potential equity

Work below the model API.

Most AI engineering roles focus on building applications around large language models. This one focuses on making the models themselves run faster, more efficiently, and at significantly lower cost.

We are partnering with an established global AI technology company to hire a Senior Principal Software Engineer to own and optimize production LLM inference systems.

You will work directly with inference runtimes, CUDA kernels, GPU memory architecture, and serving infrastructure supporting large-scale commercial applications.

This is not a traditional MLOps, model deployment, or LLM application engineering position. We need someone who understands what happens inside the inference engine and knows how to improve it.

What You'll Work On

• Profile and optimize production LLM inference workloads for latency, throughput, GPU utilization, and cost.
• Modify and improve inference runtimes such as vLLM, SGLang, and TensorRT LLM.
• Develop and optimize CUDA kernels, attention mechanisms, and GPU memory operations.
• Improve KV cache management, prefix caching, memory allocation, and inference performance under concurrent workloads.
• Optimize prefill and decode execution, including workloads with large input contexts and short generated responses.
• Explore techniques such as speculative decoding, continuous batching, quantization, and disaggregated inference.
• Identify performance bottlenecks, implement improvements, and benchmark measurable results in production.
• Collaborate with experienced AI researchers and systems engineers across a global engineering organization.

What We're Looking For

Your experience should go beyond deploying or configuring existing inference frameworks. You should have personally improved how production inference systems execute.

• Significant software engineering experience, with strong C++ programming fundamentals.
• Hands on experience developing, modifying, or optimizing CUDA or equivalent GPU kernels.
• Experience working with production LLM inference systems at meaningful scale.
• Strong understanding of GPU memory hierarchy, bandwidth, cache behavior, and compute bottlenecks.
• Experience profiling and improving latency, throughput, memory efficiency, or GPU utilization.
• Ability to navigate complex systems code, troubleshoot performance issues, and implement changes at the runtime level.
• Strong engineering judgment and the ability to explain technical decisions, tradeoffs, and measurable performance improvements.

Particularly Relevant Experience

• Deep involvement with vLLM, SGLang, TensorRT LLM, or comparable inference engines.
• Flash Attention or other attention kernel optimization.
• KV cache management, prefix reuse, and memory optimization.
• FP8, INT4, AWQ, GPTQ, or other quantization techniques.
• Prefill and decode disaggregation, speculative decoding, and inference scheduler optimization.
• NVIDIA H100 or H200 performance tuning.
• Rust development or distributed GPU inference systems.

Why Join?

Work on the problems that really move the needle. Your work will go beyond deploying models. You'll directly influence how efficiently they execute, improving the performance of commercially deployed AI systems.

Own meaningful technical challenges. Dive into GPU execution, memory behavior, inference scheduling, and runtime internals, with the scope to make substantial architectural and code-level improvements.

See measurable impact. Whether you're reducing inference latency, increasing tokens per second, or improving GPU efficiency, your engineering decisions will have tangible production outcomes.

Collaborate with exceptional technical talent. Join a globally…