Senior Software Engineer, Resilience Engineering - DGX Cloud

Responsibilities

• Build org‑wide reliability strategy, guiding how NVIDIA matures its operational practices in a 24/7 environment.
• Stand up a rigorous SLO program, defining and maintaining high standards across teams.
• Lead incident response for high‑severity incidents, ensuring low drama and high‑signal resolution.
• Build and improve production code daily, enhancing our data platform and related tooling.
• Implement chaos engineering, failure injection, and resilience testing to elevate the team’s standard practices.
• Improve standards by setting an example with your hands‑on experience and leadership.

• Deep, hands‑on experience running large‑scale production systems with a proven track record.
• Detailed understanding of failure modes in large systems, including cascading dependencies and retry storms.
• Strong software engineering skills with current, hands‑on experience in Go, Python, or similar languages.
• Proven experience establishing and maintaining an SLO program with operational rigor.
• Practical experience in reliability fields such as chaos engineering and failure injection.
• Ability to influence across team boundaries through credibility and expertise.
• 10+ years of industry experience with a Bachelor’s or Master’s degree, or equivalent experience operating systems at scale.

• Experience with in a world‑class reliability function like Google SRE or Meta production engineering.
• Expertise in operating GPU, HPC, or AI training infrastructure with outstanding failure modes.
• A track record of measurable reliability improvements within an organization.
• Proficiency with modern observability and operational tools such as Prometheus, Open Telemetry, Grafana, Pager Duty, and Rootly.

Your base salary will be determined based on your location, experience, and the pay of employees in similar positions.

Base salary range:
Level  USD to 287,500 USD. Level  USD to 356,500 USD.

You will also be eligible for equity and benefits.

NVIDIA is committed to fostering an inclusive work environment and proud to be an equal opportunity employer. We do not discriminate on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status, or any other characteristic protected by law.