Senior ML Engineer | Kimchi; Inference Stack Optimization

Great models are raw ingredients. Kimchi is what happens when you ferment them with good engineering. We started inside CAST AI helping companies run LLMs on their own Kubernetes clusters; today we run our own coding harness on any model you pick, and we run the models underneath it ourselves.
Kimchi Inference

One API line and a customer’s existing client just works:
Deep Seek, GLM, Kimi, Mini Max, Nemotron served serverless from our own GPU fleet (OpenAI- and Anthropic-compatible, pay per token, spend visible per user, team, and key). When control matters, the same models run on the customer’s own cluster instead. Behind that one line sits the machinery you’ll own: SGLang and vLLM engines, B300 nodes, a shared cluster file system that holds every checkpoint exactly once, KV caches riding coding traffic with heavy prefix reuse, and the benchmark harness that gates every change we ship.
Made by Cast AI — a Miami-headquartered, globally distributed team across 34 countries that hit unicorn status in January 2026 with a $50B+ backed investment pushing its valuation past $1 billion.

Throughput. Latency. Cold starts. Cost per token.

Move those four numbers in the right direction and two things happen: customers get faster, cheaper inference, and our margins improve. That’s the entire thesis of this role. Every loader you tune, every scheduler decision you land, every kernel you stare at shows up in a customer’s p99 and on our P&L. Your mandate is the inference stack’s optimization, whatever it takes, and the technical direction of how we optimize it is yours to argue with data.

The problem: the checkpoints run 0.3-1.5 TB, and the “right” serving configuration depends on traffic shape, sequence-length distribution, batch dynamics, GPU SKU, memory bandwidth, and quantization tolerance, all variables that shift week to week. Most teams pick a configuration once, over-provision GPUs, and absorb the cost. We don’t: we took Kimi-K3 (1.5 TB) from over an hour cold-to-serving to under twelve minutes with a shared file system, O_DIRECT weight loaders, and node-local JIT caches, and we’re nowhere near done.

Prefill/decode disaggregation and tiered KV caches are next.

Stack: Python; SGLang; vLLM;
TensorRT-LLM;
PyTorch; fastsafetensors; CUDA-adjacent tooling;
Kubernetes; virtiofs shared file systems;
Click House;
Open Telemetry;
Prometheus;
Grafana;
Git Lab CI.

• You’ve built real ML systems for 5+ years, and your portfolio shows depth in inference or training infrastructure, not just model training notebooks.
• Your Python is production-grade: services, not scripts.
• You’ve run vLLM, SGLang, or TensorRT-LLM for real, and you have a working mental model of why an engine performs the way it does on a given GPU.
• You’ve measured quantization regressions, not just compression ratios.
• Multi-GPU and multi-node setups hold no fear: tensor and expert parallelism, collective communication, and their practical failure modes.
• You know where the bytes live (page caches, O_DIRECT, network file systems under terabyte-scale reads) and how fast they move.
• You instrument before you optimize, and you can tell a real win from a benchmark artifact.
• A wide mandate excites you more than it unsettles you.

• You’ve worked on LLM routing specifically:
  Router Bench-class evals, cost/quality/latency Pareto fronts.
• Calibration, abstention, or conformal methods in production decision systems.
• You’ve run online A/B loops with drift monitoring on a model that touches real money.

• Push throughput. Continuous batching, chunked prefill, speculative decoding with draft models, CUDA-graph capture, and scheduler tuning across the engine
  fleet. Find the ceiling on each GPU SKU, then raise it.
• Cut latency. Attack…