Senior ML Systems Engineer, Inference

Runpod (the AI Developer Cloud) is hiring a full-time Senior ML Systems Engineer, Inference for a Remote

• USA role to own LLM serving performance end to end across its platform.

• Define how inference performance is measured (throughput, time to first token, inter-token latency, cost per token) and build rigorous, repeatable measurement tooling.
• Profile and diagnose performance problems across the serving stack, from scheduling and memory management down to kernels and interconnect.
• Improve serving efficiency for large, state-of-the-art models on single-node and multi-node GPU deployments.
• Turn findings into production-ready runtimes, configurations, and defaults that customers benefit from automatically.
• Work closely with product and infrastructure teams to shape how inference is offered on Runpod.
• Track the fast-moving inference ecosystem (including open source) and decide what to adopt, build, or contribute back.
• Trace bottlenecks in the serving engine/runtime and implement fixes when configuration tuning is not enough.

• 5+ years of professional system engineering experience.
• Deep, hands-on experience with vLLM, SGLang, or a comparable serving engine in production or at serious benchmark scale.
• Strong software engineering skills in Python, comfortable in large, performance-critical codebases.
• Solid understanding of LLM inference performance drivers: batching, memory, parallelism, and latency/throughput trade-offs.
• Experience with modern inference optimization techniques such as quantization, speculative decoding, or distributed serving.
• Rigor in benchmarking and performance analysis plus comfort with GPU profiling tools.
• Ability to explain results clearly in writing and turn them into decisions.

• Remote
• USA; remote does not imply worldwide eligibility.
• Must be eligible to work in the United States;
  Runpod is currently unable to sponsor employment visas.
• Full-time role.

• Experience writing or tuning GPU kernels in CUDA or Triton.
• Contributions to inference or ML systems projects.
• Experience with multi-node GPU systems and high-speed networking.
• Experience at a company where inference cost and latency were core business metrics.

• Base pay range $150,000-$220,000 (may span career levels; narrowed during interviews).
• Meaningful equity/stock options for everyone on the team.
• Generous medical, dental, and vision plans.
• Flexible PTO.
• Remote-first work with collaborative Slack-based teams.
• $1,200 home office and equipment stipend.

• The role centers on making Runpod the fastest and most cost-efficient place to run LLM inference, with end-to-end ownership of serving performance.
• Work spans measurement definition, profiling, optimization, and shipping production-ready runtime defaults.
• Scope covers models, hardware generations, single-node and multi-node GPU deployments, and customer-facing latency/cost.

Employer-published remote role. Listed location(s):
Remote

• USA. Remote does not imply worldwide eligibility. Check the original listing for residency, work authorization, time zone, language and travel requirements.

Runpod is the AI Developer Cloud. More than one million developers, from indie researchers to teams running frontier models in production, use Runpod to experiment, train, fine-tune, deploy, and scale AI on one platform. The platform has processed more than 20 billion inference requests. We closed a $100M Series A in June 2026. We're at an inflection point for AI infrastructure, and we're building the platform the next generation of developers will depend on.

We're a small, remote-first team. We take ownership seriously, move fast, and ship work that more than a million developers rely on every day. We're looking for people who care deeply, build with urgency, and want to matter at scale.

Learn more in our CEO's funding announcement:

We're looking for a ML Systems Engineer, Inference. We want Runpod to be the best place in the world to run LLM inference, meaning the fastest and the most cost-efficient. You'll lead that effort. You'll own LLM serving performance end to end. That means measuring it, understanding it, and improving it across models, hardware generations, and workloads. The work you ship will show up directly in the latency and cost our customers experience.

This is a hands-on engineering role for someone who likes finding the real bottleneck and fixing it, then turning that fix into something that runs…