Senior Storage Production Engineer - DGX Cloud

Production engineering is a field that involves crafting, building, and maintaining large-scale production systems with high efficiency and availability. It encompasses various areas, including software and systems engineering practices, storage, data management, and services. Professionals in the role of Production Engineers hold specialized knowledge and expertise across various domains, including storage architecture, high-performance distributed storage, data management, systems, networking, coding, database management, prioritization, continuous delivery and deployment, along with open-source cloud-enabling technologies such as Kubernetes, containers, and virtualization.

Their responsibilities include ensuring storage architectures are reliable, scalable, and efficient. They optimize data placement and access patterns. They manage large-scale distributed storage systems and ensure low-latency data access for HPC and AI/ML workloads.

Storage Production Engineers at NVIDIA ensure that our internal and external-facing GPU cloud services meet reliability and uptime goals as promised to the users while enabling developers to make changes to the existing system through careful preparation and planning while keeping an eye on capacity, latency, and performance. This role also requires a mindset focused on automating storage operations, improving data access efficiency, and optimizing storage performance.

Much of our software development focuses on optimizing operations through automation, enhancing system responsiveness, and improving the efficiency of storage and production systems. Since Production Engineers are responsible for the big picture of how our systems interface with each other, we use a breadth of tools and approaches to tackle a broad spectrum of challenges. Practices such as proactive storage performance monitoring, automated fault detection and remediation, scalable data redundancy methods, and integration of intelligent caching mechanisms factor into iterative improvements that are key to system reliability and efficiency.

• Design, implement, and support large-scale storage clusters, ensuring scalability, high availability, and data integrity.
• Develop and maintain storage monitoring, logging, and alerting systems to ensure proactive detection and resolution of performance issues.
• Work with AI/ML workloads to improve storage architectures for low-latency access, efficient caching, and high-throughput performance.
• Improve the lifecycle of storage services – from inception and design to deployment, operation, and continuous optimization. Support storage services before they become available through activities such as system build consulting, developing automation frameworks, capacity management, and launch reviews.
• Maintain production storage infrastructure by supervising availability, latency, and system health, leveraging predictive analytics and AI-driven automation.
• Optimize storage efficiency through compression, deduplication, tiering strategies, and intelligent workload placement.
• Scale storage systems sustainably using AI/ML-driven automation, policy-based tiering, and dynamic data migration techniques. Ensure data security and compliance by implementing encryption, access controls, and auditing mechanisms for storage systems.
• Practice sustainable incident response and blameless root cause analysis. Be part of an on-call rotation to support storage and production systems.

• BS degree or equivalent experience in Computer Science, Storage Systems, or a related technical field with 8+ years of practical experience.
• Experience with distributed and high-performance storage solutions, including clustered and parallel file systems, distributed object storage, and enterprise-grade storage systems.
• Solid understanding of block, file, and object storage technologies, including their scalability, reliability, and performance characteristics and standard processes.
• Experience with storage networking protocols such as NFS, SMB, iSCSI, S3, Fibre Channel, RDMA, and NVMe over Fabrics.
• Expertise in algorithms, data structures, complexity analysis, software…