Senior Software Engineer, Observability

Senior Software Engineer, Observability page is loaded## Senior Software Engineer, Observabilitylocations: US, CA, Santa Claratime type:
Full time posted on:
Posted Todayjob requisition 's Observability team is seeking a Senior/Staff Engineer to compose and build the next-generation, multi-region observability platform. This platform powers our rapidly expanding AI, Data, and Observability ecosystem, operating at an immense scale: trillions of metrics, hundreds of terabytes of logs, and billions of distributed traces are processed daily across high-performance datacenters and multi-cloud environments. This is a high-impact, architecture-heavy, and code-first role.

You will be responsible for architecting, building, and operating NVIDIA’s unified observability stack, which encompasses metrics, logs, traces, profiles, and analytics powered by advanced technology. Your ownership will span the entire telemetry pipeline, including ingestion, storage, query routing, governance, multi-tenant isolation, GPU-accelerated analytics, and real-time insights.

You will play a crucial role in crafting NVIDIA's global observability strategy by collaborating with teams with varied strengths including GPU Compute, Distributed Systems, Networking, ML Infra, AI Platform, and Cloud Services to ensure engineers have deep access to system health, performance, and debugging signals.
** What you’ll be doing:
*** Designing and operating scalable observability systems (metrics, logging, tracing) across multi-datacenter Kubernetes environments
* Architecting end-to-end observability pipelines, including ingestion, storage, querying, and visualization
* Extending monitoring and alerting with Prometheus, Alert manager, Thanos/Mimir, Grafana, and Open Telemetry
* Building scalable log collection and processing pipelines with Fluent Bit, Vector, Loki, or ELK/Open search stacks
* Implementing distributed tracing platforms (Tempo, Jaeger, Open Telemetry) and integrating with service meshes, load balancers, and APIs
* Defining and driving adoption of SLOs, SLIs, and error budgets across services and teams
* Automating provisioning and scaling of observability infrastructure with Kubernetes, Terraform, and custom tooling (Go, Python)
* Ensuring reliability and cost efficiency of telemetry pipelines while supporting high-volume workloads (AI/ML, HPC clusters, GPU infrastructure)
* Embedding security guidelines into observability platforms, including RBAC, TLS, secret management, and multi-tenant access controls
* Mentoring engineers and shaping Nvidia observability strategy and technical roadmap
** What we need to see:
*** 8+ years of experience with distributed systems, with a focus on observability and monitoring systems
* BS or MS in EE, ECE, CS, or equivalent experience.
* Deep expertise with metrics systems (Prometheus, Thanos, Mimir, Cortex), logging pipelines (Fluent Bit, Vector, Loki, ELK/Open search), and tracing platforms (Jaeger, Tempo, Open Telemetry)
* Strong programming skills in Go or Python for automation, operators, and custom integrations
* Experience running observability platforms on Kubernetes and operating them at scale across multi-datacenter environments
* Demonstrated skill in crafting, optimizing, and scaling telemetry pipelines handling high cardinality and high efficiency data
* Solid understanding of distributed systems, performance engineering, and debugging complex workloads
* Familiarity with service meshes, networking, and workload instrumentation (Envoy, Istio, Open Telemetry SDKs)
* Collaboration skills and the ability to influence engineering teams to adopt observability guidelines
** Ways to stand out from the crowd:
*** Strong background in software design and system architecture with excellent coding skills in Go and Python for building telemetry pipelines and microservices.
* Experience operating large-scale, multi-region observability platforms (metrics, logs, traces).
* Proven track record to meet strict SLAs for high-throughput, high-cardinality distributed systems.
* Deep expertise in data platforms and telemetry ingestion, storage, and querying layers.
* Demonstrated success collaborating across teams to drive…