Inference Engineer

Location:Mountain View / South Bay, CA

Work Arrangement:Onsite

An innovative AI infrastructure startup is seeking a Founding Engineer specializing in LLM infrastructure and inference to help build next-generation AI serving systems.

The company is developing intelligent software that analyzes AI training and inference workloads to improve performance, power efficiency, and infrastructure costs.

This is a highly hands‑on engineering opportunity focused on designing, building, and optimizing distributed LLM serving infrastructure for large language models and agentic AI workloads.

The founding engineer will take significant ownership of system architecture and implementation, working directly with the founding team to develop the company's core technology. Opportunities are also available for exceptional early‑career engineers with strong computer science fundamentals and demonstrated technical ability.

• Design, build, and optimize scalable infrastructure for hosting and serving large language models across multiple machines.
• Own architecture and implementation of distributed LLM inference systems, with significant technical autonomy.
• Develop infrastructure supporting agentic AI workloads and evolving inference requirements.
• Analyze system performance across compute, memory, and distributed execution.
• Investigate bottlenecks affecting inference latency, throughput, hardware utilization, and overall efficiency.
• Optimize inference serving strategies, including batching, scheduling, caching, and parallel execution.
• Develop performance measurement and benchmarking methodologies for representative LLM workloads.
• Evaluate and improve KV cache management, memory utilization, and inference execution efficiency.
• Collaborate with performance and infrastructure engineers to translate workload‑level optimizations into system‑wide improvements.
• Work directly with the founding team to influence architecture, engineering decisions, and development of the serving platform.

• Hands‑on experience building, implementing, or optimizing LLM inference or model‑serving infrastructure.
• Strong understanding of distributed systems and multi‑machine execution.
• Understanding of how model execution interacts with compute, memory, and machine‑level constraints.
• Ability to identify performance bottlenecks and implement measurable improvements in latency, throughput, or hardware utilization.
• Strong software engineering fundamentals and experience debugging complex systems.
• Ability to explain architectural decisions, technical trade‑offs, and measured performance improvements.
• Willingness to work across infrastructure, inference engines, runtime behavior, and system optimization in an early‑stage engineering environment.
• For founding‑level candidates, demonstrated ability to independently design and build substantial systems.

Strong computer science fundamentals, substantial engineering projects, and demonstrated ability to learn complex technical concepts quickly.

• Experience developing or contributing to LLM serving frameworks such as vLLM, SGLang, TensorRT‑LLM, or llm‑d.
• Understanding of inference optimization techniques, including KV cache management, continuous batching, Paged Attention, and Flash Attention.
• Experience optimizing inference latency, including time to first token (TTFT), throughput, and serving efficiency.
• Familiarity with distributed execution techniques such as tensor parallelism, pipeline parallelism, and expert parallelism.
• Experience with distributed machine learning frameworks such as Deep Speed, Megatron, or FSDP.
• Knowledge of GPU performance profiling, memory…