HPC & AI Management Team Lead; HPC Engineer

This position is open for external candidates only to apply.

Join the High Performance Computing Environments Group (HPC-ENV) at Los Alamos National Laboratory to support the computing infrastructure that enables scientific modeling and simulation, artificial intelligence and machine learning, and emerging computing technologies. As a member of a small team, you will play a direct role in operating production systems, managing HPC scheduling and resource allocation, and improving system reliability and efficiency.

We are a small team, and each member plays a direct role in operating our production systems. We are seeking an experienced professional in HPC scheduling and resource management to lead the team in ensuring the reliable operation of our production systems, which include:

• Large-Scale Modeling & Simulation (e.g., physics, engineering, climate) infrastructure
• Artificial Intelligence & Machine Learning (AI/ML) deployments
• Emerging Architectures for next-generation computing needs

This position will be filled at either the HPC Engineer 2 or HPC Engineer 3 level, depending on the skills of the selected candidate. Additional job responsibilities, outlined below, will be assigned if the candidate is hired at the higher level.

• Configure, deploy, maintain, and troubleshoot HPC job scheduling software and associated databases on existing and new production systems.
• Analyze scheduling performance and optimize resource allocation for modeling and simulation, AI/ML, and other computing workloads.
• Develop and maintain workflow automation, continuous integration and continuous delivery (CI/CD) pipelines, and Infrastructure as Code (IaC) solutions for testing, provisioning, configuration, and deployment.
• Collaborate with vendors and internal technical teams to resolve issues, implement updates, and integrate scheduling solutions with new computing architectures.
• Provide technical direction, coordinate project activities and task execution, and provide input to group management during annual performance reviews.
• Participate in an on-call rotation to support production operations.

• Lead the deployment and integration of new HPC systems and architectures, including automated provisioning and configuration.
• Lead efforts to optimize scheduling and resource utilization across large-scale simulation and AI/ML workloads.
• Work with stakeholders and vendors to define system requirements and guide technical solutions.
• Evaluate and introduce new approaches to HPC system management, automation, and orchestration.
• Partner with production system administrators to diagnose and resolve complex system issues affecting applications running on HPC resources.

Minimum

Job Requirements:

• Linux Administration and Dev Ops:
  Demonstrated experience administering Linux systems and applying Dev Ops practices to support reliable system operations.
• Automation and Programming:
  Experience automating technical workflows and programming or scripting in languages such as Python, Bash, C, or C++.
• Computing Environments:
  Working knowledge of one or more of the following: large-scale system administration; job scheduling and modeling and simulation workflows; or AI/ML infrastructure, deployment, and tooling.
• Technical Direction and
  
  Collaboration:
  
  Ability to provide technical direction, coordinate work, and collaborate with team members, vendors, and internal partners to resolve technical issues and support production operations.

Additional Job Requirements for HPC Engineer 3:

• Production HPC Operations:
  Demonstrated experience managing production HPC environments and resolving complex operational issues.
• Technical Leadership:
  Demonstrated accomplishments…