Senior Software Engineer, Inference San Francisco, CA NY | Seattle, WA

Staff + Senior Software Engineer, Inference About Anthropic

Anthropic’s mission is to create reliable, interpretable, and steerable AI systems. The team is a rapidly growing group of researchers, engineers, policy experts, and business leaders working together to build beneficial AI systems.

The Inference team is responsible for building and maintaining the critical systems that serve Claude to millions of users worldwide. The role spans the entire stack, from intelligent request routing to fleet-wide orchestration across diverse AI accelerators, with a dual mandate of maximizing compute efficiency and enabling breakthrough research.

• Design, build, and maintain distributed systems serving Claude worldwide.
• Develop intelligent request routing, load balancing, and traffic management across thousands of accelerators.
• Maximize compute efficiency through autoscaling and orchestration of production, research, and experimental workloads.
• Build and operate production‑grade deployment pipelines for releasing new models.
• Provide high‑performance inference infrastructure for researchers to develop next‑generation models.
• Integrate new AI accelerator platforms and support inference for new model architectures.
• Use observability data to tune and improve performance based on real‑world production workloads.
• Demonstrate significant software engineering experience, particularly with distributed systems.
• Results‑oriented, with a bias toward flexibility and impact.
• Willingness to take on tasks outside the job description.
• Enjoy pair programming.
• Desire to learn more about machine learning systems and infrastructure.
• Thrive in environments where technical excellence directly drives business results and research breakthroughs.
• Care about the societal impacts of your work.

• Experience with high‑performance, large‑scale distributed systems.
• Experience implementing and deploying machine learning systems at scale.
• Experience with load balancing, request routing, or traffic management systems.
• Familiarity with LLM inference optimization, batching, and caching strategies.
• Experience with Kubernetes and cloud infrastructure (AWS, GCP, Azure).
• Proficiency in Python or Rust.

• Designing intelligent routing algorithms that optimize request distribution across thousands of accelerators.
• Autoscaling the compute fleet to match supply with demand across production, research, and experimental workloads.
• Building production‑grade deployment pipelines for releasing new models to millions of users.
• Integrating new AI accelerator platforms to maintain a hardware‑agnostic competitive advantage.
• Contributing to new inference features (e.g., structured sampling, prompt caching).
• Analyzing observability data to tune performance based on real‑world production workloads.
• Managing multi‑region deployments and geographic routing for global customers.

$320,000 – $485,000 USD per year.

Minimum education: Bachelor’s degree or equivalent combination of education, training, and/or experience.

Required field of study: A field relevant to the role, demonstrated through coursework, training, or professional experience.

Minimum years of experience: Years of experience required will correlate with the internal job level requirements for the position.

Location-based hybrid policy: Staff are expected to work from one of our offices at least 25% of the time.

Visa sponsorship: We sponsor visas for qualified candidates.

As set forth in Anthropic’s Equal Employment Opportunity policy, we do not discriminate on the basis of any protected group status under any applicable law.