Principal Infrastructure Engineer, AI Cluster & Validation

Set the technical bar for production readiness and performance across very large AI and HPC clusters. This principal-level role combines infrastructure architecture, hands-on workload testing, deep failure analysis, and software tooling to improve reliability and delivered compute capacity.

• Define cluster health criteria, performance targets, and acceptance processes for clusters containing thousands of GPUs, including before and after customer handover.
• Run distributed training and inference workloads at scale, use their results to identify fleet-wide issues, and turn findings into engineering improvements.
• Investigate slow or stalled jobs and cluster regressions across accelerators, firmware, host topology, high-speed networking, storage, schedulers, and communication libraries.
• Build automated qualification and burn-in pipelines for nodes, racks, and clusters, covering benchmarks, collective communication, thermal and power behavior, and link stability.
• Tune networking, communication libraries, storage paths, and host settings to improve usable workload throughput.
• Develop Python tools for triage, telemetry correlation, and regression detection; establish validation and observability practices; and mentor teams through runbooks and incident reviews.
• Work with infrastructure, platform, reliability, and customer-facing teams to incorporate operational needs into provisioning and remediation workflows.

• Bachelor’s degree or equivalent practical experience in a relevant technical field.
• At least 10 years building, operating, or debugging large-scale compute infrastructure, including substantial staff- or principal-level technical leadership across teams.
• Hands-on experience running large-scale AI training or inference, with knowledge of distributed parallelism and frameworks such as PyTorch, Megatron-LM, or Deep Speed.
• Experience qualifying large GPU clusters and defending benchmark methods and results to technical and customer audiences.
• Strong skills diagnosing distributed performance problems with tools such as NCCL tracing, Nsight, PyTorch Profiler, perf, or fabric telemetry.
• Deep knowledge of Infini Band or RoCE, RDMA, Linux systems behavior, and production Python; experience with C/C++ or Go and SLURM or Kubernetes at scale.

• A graduate degree; experience bringing a new GPU supercluster into production; familiarity with NVIDIA or AMD accelerator stacks, parallel storage, observability platforms, infrastructure-as-code, and bare-metal management systems; or published benchmark and scaling work.