Senior Backend Engineer; Inference Platform

• Every model call from our products and customers passes through one layer: our inference gateway. It allows users to select a model and we intelligently route the call to optimize on cost , latency and high availability. We are now building the control plane above it, to manage and govern access, performance and cost
• You will own and scale that gateway. The work sits between model serving and product: routing and fallback across hosted providers and our own models, cache-aware routing decisions, metering accurate enough to bill on, and enforcement on the request path within a single-digit-millisecond budget
• Own the inference gateway that every model call routes through, including provider and model selection, fallback, and caps
• Build intelligence layer to autonomously pick the right model for the right turn in the task
• Make routing decisions that account for prefix-cache behavior, batching, and time to first token, not only price and availability
• Build cost attribution and metering across providers, including cached-token accounting, accurate enough to bill and enforce on
• Build request-path enforcement that degrades gracefully and never takes down a customer’s inference call
• Ship inference-time changes in shadow mode first, with the measurement to prove they work

• A remote-first work environment with hubs in California, NYC, and Canada that offer monthly in-person gatherings
• Unlimited PTO with 11 U.S. holidays observed and a week shutdown in December to rest and recharge
• Chance to collaborate with a team at the forefront of AI research
• 12 weeks of paid paternity leave in the US; additional time off is also considered
• In-person coworking weeks 1 to 2 times a year
• 401k program, 3% match coming January 2025
• Competitive health insurance plan, where 100% of the policyholder is covered and 75% for dependents
• $1,200 / year Health & Wellness Allowance to support your personal goals
• $500 work-from-home stipend to be used up to a year of your start date

Experience building or operating a proxy, gateway, or data plane on the critical path of production traffic, with sound instincts for failing openA strong backend generalist (Python, Kubernetes). We expect you to pick up whatever you have not touched yet Hands-on production experience with model serving on stacks such as vLLM, SGLang (or similar platforms), with a working understanding of continuous batching, KV and prefix caching, and the tradeoff between time to first token and throughput

Fluency across provider APIs: tokenization, prompt-caching semantics, and the differences in how usage and cached tokens are reported and priced

You have tuned a serving stack against live traffic and know what moves p99 in practice, not only in theory

We weigh trajectory and recent growth more heavily than years of experience

Comfort with streaming responses and the parts that get awkward: partial state, mid-stream failures, and work that can only happen once the stream closes

Usage metering, billing, or cost optimization at scale

Experience at an inference provider or GPU cloud, or contributions to an open-source serving stack

Fine-tuning and evaluating models in production