AI Infrastructure Engineer

Location: Livermore, CA — Hybrid, with regular onsite work at our data center facility

Job Type: Full-Time with Amaara Networks

Experience Level: 4–7 years

Travel: Occasional travel, up to 10–15%

Amaara Networks builds and operates the physical and systems infrastructure that supports modern AI workloads. Our teams deploy, configure, maintain, and troubleshoot high-performance GPU clusters and the networking infrastructure that enables customers to train and run AI models.

As an AI Infrastructure Engineer, you will support the full lifecycle of high-performance GPU infrastructure—from physical installation and commissioning through Linux configuration, GPU validation, networking, troubleshooting, and production operations. You will be responsible for resolving complex infrastructure issues, identifying root causes, working directly with vendors and customers, and developing the documentation and automation that makes our infrastructure more reliable and scalable.

This role is ideal for someone who enjoys both working in the data center and solving problems from the terminal.

• Infrastructure Troubleshooting & Escalation:
  Own Tier 2 and Tier 3 infrastructure escalations from investigation through resolution. Perform root-cause analysis on server and infrastructure failures using BMC, IPMI logs, kernel messages, MCE, EDAC records, thermal and power telemetry, and vendor diagnostics. Identify recurring hardware and infrastructure issues and develop permanent corrective actions. Participate in post-incident reviews and translate findings into monitoring improvements, runbooks, and operational standards.
  
  Work directly with hardware vendors on diagnostics, RMAs, firmware issues, and field-service activities. Communicate clearly with customers during incidents and scheduled maintenance windows. Coordinate remote hands and field technicians during work performed at remote sites. Participate in an infrastructure on-call rotation.
• GPU Infrastructure:
  Deploy, administer, and troubleshoot high-performance GPU compute systems. Support NVIDIA H100, H200, B200, B300 SXM and PCIe platforms, AMD Instinct, or comparable accelerator platforms. Use tools such as nvidia-smi, DCGM, and DCGM diagnostics to troubleshoot GPU health and performance issues. Analyze XID errors, ECC events, NVLink, NVSwitch health, thermal conditions, and power-related throttling. Install and maintain GPU drivers, CUDA toolkits, Fabric Manager, and container runtimes.
  
  Manage software and driver compatibility across the GPU fleet. Perform GPU stress testing, burn-in, NCCL testing, bandwidth testing, and all-reduce benchmarking. Investigate performance issues including straggler nodes, reduced collective bandwidth, PCIe degradation, NUMA configuration, and topology problems.
• High-Performance Networking:
  Configure and support Infini Band and RoCEv2 high-performance AI networking environments. Configure switches, manage firmware, and validate network topology. Support Infini Band subnet management, partitioning, routing, and topology validation. Use tools such as UFM, OFED, ibdiagnet, ibnetdiscover, and perfquery to troubleshoot network issues. Diagnose link flaps, symbol errors, degraded optics, cabling issues, and port-level failures. Configure and troubleshoot RoCE environments, including PFC, ECN, lossless queues, and congestion control.
  
  Administer RDMA, OFED stacks, NIC, HCA firmware, bonding, VLANs, and management networks. Support enterprise networking, including L2, L3 networking, VLANs, LACP, MLAG, VLT, firewalls, NAT, VPN, DHCP, and DNS. Validate network performance following deployments and infrastructure changes. Maintain accurate network topology, port maps, and…