HPC Operations Engineer

We have an opening for an HPC Operations Engineer to develop, deploy, and maintain the tools, automation, and processes that keep Livermore Computing high-performance computing systems available and performing at high levels of reliability and performance. You will work in a challenging, team-oriented environment, applying technical knowledge of HPC systems and software development to engineer monitoring, alerting, and diagnostic solutions for Livermore Computing’s (LC) high-performance computing clusters, storage systems, networks, and data center facilities.

You will contribute to technical projects as a member of a technical team, and collaborate with system administrators, software developers, and facilities engineers to improve the reliability and efficiency of LC operations. This position is in the Livermore Computing Division within the Computing Directorate.

This position will be filled at either level based on knowledge and related experience as assessed by the hiring team. Additional job responsibilities (outlined below) will be assigned if hired at the higher level.

This position requires full-time on-site presence due to the nature of the work.

Join us and make YOUR mark on the World!

Lawrence Livermore National Laboratory (LLNL) has turned bold ideas into world-changing impact advancing science and technology to strengthen U.S. security and promote global stability.

Our mission spans four critical national security areas nuclear deterrence, threat preparedness, energy security, and multi-domain defense empowering teams to take on the toughest challenges of today and tomorrow. With a culture built on innovation and operational excellence, LLNL is a place where your expertise can make a real impact.

You will

• Develop, deploy, configure, and maintain monitoring, alerting, and observability tools for LC HPC clusters, storage systems, high-speed networks, and data center facility infrastructure.
• Create and maintain software tools and processes that streamline alarm response, diagnostics, and routine operational tasks using established methods and procedures.
• Configure and integrate data from building management and facility systems, such as power and cooling, with HPC system monitoring to provide a unified operational view under established technical direction.
• Share alarm response and monitoring duties with operations staff, and analyze, diagnose, and resolve hardware, software, network, and facility problems of limited complexity, determine root cause, and implement corrective actions in collaboration with system administrators, vendors, and facilities staff.
• Participate in the deployment, testing, and validation of new HPC systems and components, and the decommissioning of retired systems.
• Develop and maintain technical documentation, procedures, and runbooks for operational tools and processes.
• Respond to customer problems and questions, working with Hotline, system administration, and other LC groups to improve the quality of service for end users.
• Perform other duties as assigned.

• Design and develop moderately complex software tools and automations that improve the operation, reliability, and efficiency of LC computing and facility systems.
• Analyze system, alarm, and incident data to identify trends, and implement moderately complex strategies to improve monitoring coverage, alarm accuracy, and operational processes.
• Lead discrete technical projects or work packages, such as the deployment, validation, and decommissioning of HPC systems, coordinating tasks with system administration, facilities, and other LC groups.
• Diagnose and resolve moderately complex to complex system and facility problems and serve as a technical…