Senior Systems Software Engineer - Fleet Debuggability

NVIDIA’s invention of the GPU in 1999 sparked the growth of the PC gaming market, redefined modern computer graphics, and revolutionized parallel computing. More recently, GPU deep learning ignited modern deep learning — the next era of computing — with the GPU acting as the brain of computers, robots, and self-driving cars that can perceive and understand the world. Today, NVIDIA is increasingly known as "the AI computing company."

We are the Datacenter System Software team and are looking for a highly motivated, creative Senior Engineer to drive Fleet‑Scale Debuggability end to end. The role focuses on designing, architecting, and building infrastructure, tooling, and analytics to collect multi‑rack‑scale logs. Solutions must normalize, correlate, and reason over logs from multiple components, trays, or racks including NVIDIA’s GPUs, CPUs, and Network products, fetched in‑band or out‑of‑band to help triage fleet‑level issues seen by our customers.

Your work directly shortens the path from a raw, noisy log stream to actionable root causes.

• Architect, design, and build fleet‑wide log collection and analysis solutions that aggregate signals across components, trays, and racks.
• Develop tooling to collect, normalize, and time‑align logs from heterogeneous sources — kernel and driver logs, syslog, Redfish event logs, SEL, firmware and BMC logs — over both in‑band and out‑of‑band channels.
• Build and maintain a log catalog and taxonomy that maps raw log signatures to fault classes, severity, and remediation guidance, making triage repeatable rather than tribal knowledge.
• Develop debug and root‑cause tooling that turns high‑volume fleet logs into ranked, actionable diagnoses for hardware, firmware, and platform faults.
• Drive the design for collecting and analyzing logs at fleet scale while keeping overhead on production compute nodes low.
• Partner with all matrixed organizations—developers, SWQA, and product engineering—in a fast‑moving environment with end‑to‑end logging solutions, event schemas, and the contract between log producers and your tooling.
• Steward the project’s open‑source release: keep internal and public code paths clean, review community contributions, and represent the tooling in upstream discussions.
• Write design docs and own end‑to‑end delivery, working across teams from definition through implementation, debugging, testing, and early customer support.
• Perform code reviews and partner with development and QA to strengthen unit testing, integration coverage, and test plans.
• Track work through Jira and bug‑management tools and build a realistic end‑to‑end execution plan in collaboration with other engineers and managers.

• 10+ years in the software industry with specialization in system software and/or firmware development.
• BS, MS, or PhD in CS, CE, EE, or a related technical field — or equivalent experience.
• Proven track record of shipping scalable server products or fleet‑wide experience.
• A self‑starter who loves finding creative solutions to complicated problems, with excellent written and oral communication skills—including executive‑level reporting—strong work ethic, and dedication to teamwork.
• Flexibility to work and communicate effectively across teams, partners, and time zones.
• Experience with SCM (e.g., Git, Perforce) and project‑management tools like Jira.
• Strong, demonstrable skills in Python or RUST.
• Deep Linux systems experience: kernel and driver logs, syslog, journald, and the realities of debugging on server platforms.
• Hands‑on experience with out‑of‑band management and platform interfaces—BMC, Redfish, IPMI, SEL—and an understanding of in‑band vs. out‑of‑band trade‑offs.
• Strong skills in log parsing, normalization, and structured logging, and comfort designing schemas and taxonomies for machine‑readable events.

• Experience leading debuggability solutions on sophisticated rack‑scale compute architectures like GB200/GB300 NVL
  72.
• Familiarity with log and telemetry analytics stacks (e.g., Open Search/ELK, Loki, Prometheus, Grafana, Pager Duty) and time‑series databases.
• Hands‑on experience with x86/ARM system architecture and coding (C/C++,…