Data Scientist and AIOps Team Leader; HPC Engineer

Position: Data Scientist and AIOps Team Leader (HPC Engineer 3)
** What You Will Do*
* ** This position is open for external candidates only to apply.*
* The High Performance Computing (HPC) Division at Los Alamos National Laboratory provides scientific computing resources consisting of some of the largest HPC systems in the world as well as numerous large commodity clusters.  Our HPC Systems Group (HPC-SYS) is seeking a Team Leader for the  AIOps team to make better use of our vast amounts of HPC System and Application data, this team will add AI solutions to our HPC infrastructure.  

The new team will work alongside the other Teams in HPC-SYS, Monitoring, Web Services and Cybersecurity.  The Monitoring Team is responsible for collecting all data from our Data Centers, everything from Facilities to Clusters, and implementing operational dashboard, alerts and reports using tools like Splunk.  Our Web Servers team runs our admin and user facing web sites, including user Documentation, Ticketing systems and Gitlab.  

Our Cybersecurity Team monitors and implements cybersecurity policies on our HPC systems.

The AIOps team will have three major focus areas, LLMs, System Data Analysis and System Automation.  We have a large set of HPC specific documentation for both users and admins that will be integrated into LLMs, this team will be responsible for designing, building, and running the LLMs, as well as adding agentic capabilities to increase the usefulness to the user.  

We have massive amounts of system data. This team will implement ML and Data Science techniques to perform deeper analysis of the data to improve performance analysis, event correlation and anomaly detection.  Finally, the team will investigate AI driven workflows for task automation within the Data Center.  You will work closely with members of the AIOps team and System Matter Experts (SMEs) in different HPC areas to design and develop these tools using our on-prem analysis and AI systems.

The successful candidates' scope will include monitoring and analyzing system performance to identify anomalies.  You will analyze large volumes of data to identify patterns and trends using ML and Data Science techniques with the goal of developing automation scripts and workflows to implement proactive measures.  You and the team will maintain the AIOps platforms and tools including the user-facing LLMs.  The successful candidate will continue actively growing their technical skills and keeping up to date with the latest technologies in the field.  

In addition, the selected candidate will have the opportunity to develop technical products such as technical documentation, presentations, technical papers, and reports, to communicate findings internally and at conferences.  This position is full-time and is located at Los Alamos National Laboratory in Los Alamos, New Mexico.

** What You Need*
* ** Minimum

Job Requirements:

*
* +  
** HPC Systems & Linux Administration:
** Strong understanding of compute nodes, schedulers, high-speed networks, parallel file systems, accelerators, containers, and common HPC workloads. Proficient with the Linux command line and hardware and software security practices.

+  
** Technical Leadership & AIOps Strategy:
** Experience leading multidisciplinary teams of HPC engineers, data scientists, ML engineers, and developers. Able to define and execute a roadmap for applying AI and ML to improve reliability, efficiency, performance, and user support.

+  
** Python & Software Development:
** Strong programming skills with Python and AI frameworks for data analysis, model development, automation, APIs, testing, and production-quality services, including Git-based version control.

+  
** Machine Learning & Statistical Analysis:
** Strong statistical skills and working knowledge of ML algorithms for anomaly detection, classification, clustering, forecasting, time-series analysis, event correlation, and predictive maintenance.

+  
** Monitoring & Observability:
** Extensive experience analyzing system logs, telemetry, metrics, and traces, along with alerting and observability platforms.

+  
** Root Cause & Performance Analysis:
** Able to correlate failures and anomalous behavior across compute, storage, network, scheduler, facility, and application data. Experienced in identifying bottlenecks, resource contention, and inefficient workloads.

+  
** Data Engineering:
** Able to design pipelines that collect, normalize, enrich, store, and retrieve large volumes of HPC telemetry and operational data.

+  
** MCP & Systems Integration:
**…