Distinguished Engineer, Scaled Inferencing
Job in
Charlotte, Mecklenburg County, North Carolina, 28201, USA
Listing for:
Bazeta
Full Time
position
Listed on 2026-11-16
Job specializations:
-
Software Development
Job Description & How to Apply Below
# Distinguished Engineer, Scaled Out Inferencing United States
46 minutes agoID 5136756
Price on request## Details
Employment type:
Full-time Remote:
Yes Company:
NVIDIA## Description NVIDIA is leading the way in groundbreaking developments in Artificial Intelligence, High-Performance Computing and Visualization. The GPU, our invention, serves as the visual cortex of modern computers and is at the heart of our products and services.
As a technology leader at NVIDIA, you will lead the development of our global strategy for scaled-out AI inferencing. You will architect the high-throughput, low-latency distributed pipelines and model serving strategies required for massive scale and production reliability. You will define and drive the technical roadmap for full-lifecycle, from deployment and versioning to automated scaling, across enterprise and cloud environments. Working with NVIDIA leadership, you will establish the systems and orchestration layers that enable the world’s most advanced AI models to run with peak performance and efficiency.
This is a high-impact role where your technical decisions will influence the trajectory of AI systems globally.
What You’ll Be Doing
• Lead the architecture and development of distributed, high-performance inferencing pipelines for large-scale AI model serving across enterprise and cloud environments.
• Define the technical strategy for model deployment, versioning, and scaling to support hundreds of thousands of requests per second with low latency and high reliability.
• Drive hardware-software co-optimization, ensuring that NVIDIA’s AI stack—including Dynamo and TensorRT-LLM—delivers peak performance at scale.
• Guide open-source efforts and community collaboration around Dynamo, TensorRT-LLM, and inference-related projects, shaping industry standards.
• Develop and maintain relationships with key stakeholders, including product managers, hardware engineers, and research teams, to align inferencing strategy with broader business goals.
• Stay at the forefront of inferencing technologies, evaluating emerging techniques and integrating them into NVIDIA’s offerings.
What We Need to See
• MS or PhD in Computer Science, Electrical Engineering, or a related field (or equivalent experience).
• 15+ years of experience in distributed systems, cloud infrastructure, or AI/ML engineering, with a focus on large-scale model serving and inference.
• Deep expertise in AI inferencing frameworks, including TensorRT-LLM, vLLM, and other model serving technologies.
• Strong understanding of GPU architecture, CUDA programming, and performance optimization techniques.
• Proven track record of leading cross-functional teams and driving technical strategy at an organizational level.
• Experience with Kubernetes, Docker, and cloud-native technologies for deploying and scaling AI workloads.
• Excellent communication skills and the ability to influence technical direction across the company.
Ways to Stand Out From the Crowd
• Contributions to open-source inference projects, such as Dynamo, TensorRT-LLM, or vLLM.
• Experience with large-scale AI deployments in production environments, including multi-region and multi-cloud setups.
• Deep knowledge of model optimization techniques, including quantization, pruning, and distillation.
• Published research or patents in AI inference, distributed systems, or related fields.
NVIDIA is leading the way in groundbreaking developments in Artificial Intelligence, High-Performance Computing and Visualization. The GPU, our invention, serves as the visual cortex of modern computers and is at the heart of our products and services.
Our work opens up new universes to explore, enables amazing creativity and discovery, and powers what were once science…
To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction