LLM Inference Frameworks and Optimization Engineer

About The Role

, we are building state‑of‑the‑art infrastructure to enable efficient and scalable inference for large language models (LLMs). Our mission is to optimize inference frameworks, algorithms, and infrastructure, pushing the boundaries of performance, scalability, and cost‑efficiency.

We are seeking an Inference Frameworks and Optimization Engineer to design, develop, and optimize distributed inference engines that support multimodal and language models s role focuses on low‑latency, high‑throughput inference, GPU/accelerator optimizations, and software‑hardware co‑design, ensuring efficient large‑scale deployment of LLMs and vision models.

• Design and develop fault‑tolerant, high‑concurrency distributed inference engine for text, image, and multimodal generation models.
• Implement and optimize distributed inference strategies, including Mixture of Experts (MoE) parallelism, tensor parallelism, pipeline parallelism for high‑performance serving.
• Apply CUDA graph optimizations, TensorRT/TRT‑LLM graph optimizations, PyTorch‑based compilation (torch.compile), and speculative decoding to enhance efficiency and scalability.
• Collaborate with hardware teams on performance bottleneck analysis and co‑optimize inference performance for GPUs, TPUs, or custom accelerators.
• Work closely with AI researchers and infrastructure engineers to develop efficient model execution plans and optimize end‑to‑end model serving pipelines.

• Must-Have
  • 3+ years of experience in deep learning inference frameworks, distributed systems, or high‑performance computing.
  • Familiarity with at least one LLM inference framework (e.g. TensorRT-LLM, vLLM, SGLang, TGI).
  • Background knowledge in GPU programming (CUDA/Triton/TensorRT), compiler, model quantization, and GPU cluster scheduling.
  • Deep understanding of KV cache systems such as Mooncake, Paged Attention, or custom in‑house variants.
  • Proficiency in Python and C++/CUDA for high‑performance deep learning inference.
  • Deep understanding of Transformer architectures and LLM/VLM/Diffusion model optimization.
  • Knowledge of inference optimization techniques such as workload scheduling, CUDA graph, compiled kernels, and efficient kernels.
  • Strong analytical problem solving skills with a performance‑driven mindset.
  • Excellent collaboration and communication skills across teams.
• Nice-to-Have
  • Experience developing software systems for large‑scale data center networks with RDMA/RoCE.
  • Familiarity with distributed file systems (e.g., 3FS, HDFS, Ceph).
  • Familiarity with open‑source distributed scheduling/orchestration frameworks such as Kubernetes (K8S).
  • Contributions to open‑source deep learning inference projects.

We offer competitive compensation, startup equity, health insurance, and other benefits. The US base salary range for this full‑time position is: $160,000 – $230,000 + equity + benefits. Compensation is determined by location, level and role.

Together AI is an Equal Opportunity Employer and offers equal employment opportunity to everyone regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity, veteran status, and more.

Please see our privacy policy at https://(Use the "Apply for this Job" box below)./privacy