Inference Acceleration Engineer Runtime Remote; US, EU
-
Software Development
Job Description & How to Apply Below
Own end-to-end performance for one or more model families on our managed inference runtime. We are 10–20× faster than hyperscaler inference today; the bar is to widen that gap on every release.
Hiring manager replies within 5 business days.
About the teamThe runtime team is the engineering side of the research lab. Six engineers today. We write the kernels, the scheduler, and the serving layer for managed inference. The runtime ships into customer production weekly.
Reports to the head of runtime. One on-call rotation shared across runtime, cluster SRE, and customer engineering.
What you'll do- 01
Profile real customer workloads with Nsight, rocprof, and our internal tracer; turn the bottleneck into a paper-quality benchmark and a merged kernel.
- 02
Write fused, attention-aware kernels in CUDA and Triton (and ROCm/HIP for AMD MI300X) targeting H100, H200, B200, and B300.
- 03
Own one model family end-to-end: tokenizer through KV cache through speculative decoding, on every supported accelerator.
- 04
Design and ship the next round of long-context primitives — paged KV, ring attention, sliding-window cache eviction.
- 05
Co-author at least one external write-up per quarter: a blog post, a paper, or a kernel released to the open ecosystem.
- 06
Carry the runtime on-call rotation alongside cluster SRE and customer engineering — about one week per six.
Five-plus years of systems-level performance work, with at least two of those years deep in GPU kernels.
Strong CUDA and at least one of Triton, CUTLASS, or HIP. Comfortable reading PTX and SASS when the profile demands it.
Track record of measurable speed-ups landed in production or in widely-used open source — bring a PR or a perf graph.
Comfort with transformer internals: attention variants, KV cache shapes, quantization (FP8, INT4 / NF4), speculative decoding.
Empirical instincts. You design experiments, hold variables fixed, and write down the numbers before the next change.
Nice to have, not requiredPublished kernels in vLLM, SGLang, TensorRT-LLM, Flash Attention, or similar.
Experience tuning collectives (NCCL / RCCL) and topology-aware scheduling.
AMD MI300X / MI325X experience — most candidates do not have this; we will weight it heavily.
Prior research experience or co-authorship on systems / ML papers.
Compensation In writing, like everything elseWe publish bands. We meet them. The number you see on the offer is the same number your future peers got at the same level. We do not negotiate; we level.
Base$220,000 – $360,000 USD (US), depending on level (IC4 / IC5 / IC6).
EquityMeaningful early-stage equity, refreshed on tenure milestones, not review cycles.
NotesPay bands are published internally and shared in the manager call. We do not negotiate; we level.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction