Inference Infrastructure Engineer, Serving
Job in
Palo Alto, Santa Clara County, California, 94306, USA
Listed on 2026-08-16
Listing for:
Jobtailor
Full Time
position Listed on 2026-08-16
Job specializations:
-
Software Development
AI Engineer (Applied/Software), Machine Learning/ ML Engineer
Job Description & How to Apply Below
- Build low-latency, high-throughput inference serving systems for our large multimodal models
- Design and implement techniques that improve latency, throughput, and efficiency, including quantization, batching, speculative decoding, and KV cache management
- Optimize our codebase and GPU fleet to fully utilize hardware FLOPs, bandwidth, and memory
- Implement multi-GPU and multi-node model parallelism for serving (tensor or pipeline parallel)
- Build autoscaling and load balancing for production ML services
- Establish standards for reliability, observability, and reproducibility across the inference stack
- Collaborate with researchers to enable high-performance inference for novel architectures
- 3+ years of experience building low-latency, high-throughput inference serving systems for large models
- Strong knowledge of inference optimization techniques (quantization, batching, speculative decoding, KV cache management)
- Hands-on experience with serving frameworks such as vLLM, TensorRT-LLM, Triton, or SGLang
- Experience with multi-GPU/multi-node model parallelism for serving (tensor or pipeline parallel)
- Strong systems programming skills; C++/CUDA a plus alongside Python
- Experience with autoscaling and load balancing for production ML services
- A track record of GPU cost optimization at scale
- Preferred qualifications (strong candidates may have some, not all):
Experience serving multimodal (vision + language) models - Contributions to open-source ML or systems infrastructure projects (e.g., vLLM, SGLang, TensorRT-LLM, Triton)
To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
Search for further Jobs Here:
×