Senior Director, Inference Products and Optimizations

Senior Director of Engineering – Inference Engine organization at Digital Ocean

• Recruit, mentor, and coach engineers to foster a culture of ownership, technical excellence, and continuous improvement.
• Work with product teams to define and execute on the roadmap for Digital Ocean’s Inference Products, including Serverless, Dedicated, Inference Router, Batch, and Multimodal Inference.
• Lead the design and evolution of the inference serving stack, driving technical strategy across vLLM, SGLang, and LLM‑D to optimize throughput, latency, and GPU utilization at scale.
• Architect the model‑serving and optimization layer, spanning quantization, KV‑cache management, speculative decoding, and disaggregated serving, to deliver best‑in‑class performance‑per‑dollar.
• Collaborate with product management, other engineering teams, and key stakeholders to align priorities, manage dependencies, and communicate progress and risks.
• Ensure production health, stability, and on‑call rotation to maintain customer SLAs.
• Institutionalize benchmarking frameworks, observability, and auto‑tuning capabilities to guide system and infrastructure tuning efforts.
• Encourage contributions to open‑source inference engines to advance capabilities.

• 10+ years of software engineering experience with 6+ years in technical leadership or management, ideally in inference or AI/ML systems.
• Deep expertise in distributed systems design, modern AI/ML technologies, Kubernetes at scale, LLM inference, and AI workload orchestration.
• Strategic knowledge of GPU architectures (NVIDIA/AMD), interconnects, and hardware topology for AI training and inference performance.
• Familiarity with container runtime internals, system isolation, and security contexts for shared infrastructure risk management.
• Expertise in defining, tracking, and operationalizing deep infrastructure and inference metrics (e.g., TTFT, TPOT) for performance improvements and SLO compliance.
• Product mindset: ability to translate complex technical requirements into user‑focused product features while balancing innovation and reliability.
• Excellent communication skills for explaining technical decisions to non‑technical stakeholders and aligning diverse teams.
• Strong ownership mentality and proactive drive to identify and resolve issues preventing delivery of value.

Compensation: $274,400 – $343,000

Hybrid role