Site Reliability Engineer

About this position

The National Energy Research Scientific Computing Center (NERSC) is inviting applications for the position of Site Reliability Engineer. NERSC’s mission is to accelerate scientific discovery through high performance computing and data analysis for the DOE Office of Science programs. NERSC provides critical HPC and data systems and support for NERSC’s 11,000 plus users researching energy, physics, materials science, and chemistry and other DOE mission areas.

As a Site Reliability Engineer in the Operations Technology Group, you will be a member of a 24x7 team that helps ensure NERSC is accessible, reliable, and secure for our scientific users. This position leverages advanced data collection and monitoring systems to proactively manage the health of our environment. Ultimately, your work ensures that NERSC’s computational power remains an uninterrupted catalyst supporting fundamental scientific research related to energy.

The National Energy Research Scientific Computing Center (NERSC) is inviting applications for the position of Site Reliability Engineer. NERSC’s mission is to accelerate scientific discovery through high performance computing and data analysis for the DOE Office of Science programs. NERSC provides critical HPC and data systems and support for NERSC’s 11,000 plus users researching energy, physics, materials science, and chemistry and other DOE mission areas.

As a Site Reliability Engineer in the Operations Technology Group, you will be a member of a 24x7 team that helps ensure NERSC is accessible, reliable, and secure for our scientific users. This position leverages advanced data collection and monitoring systems to proactively manage the health of our environment. Ultimately, your work ensures that NERSC’s computational power remains an uninterrupted catalyst supporting fundamental scientific research related to energy.

Describe the duties/functions essential to performing the job.

• Works an onsite 5-day weekly schedule consisting of Owl (midnight-8 am) shifts to monitor the NERSC HPC Facility.
• Review and respond to alerts from computer systems, storage, network, and other data center/facility-related systems by triaging or calling the appropriate on‑call staff.
• Create appropriate solutions to improve processes, prevent issue recurrence, and automate responses to all routine service conditions.
• Identify issues and propose solutions that will improve monitoring capabilities or provide better automation for triage.
• Possess expertise in Service Now and its usage to develop and implement customized service management solutions.
• Respond to alerts from multiple systems to ensure that data collection continues 24/7, providing real time information for diagnoses.
• Develop and maintain tools within the monitoring pipeline in collaboration with the Operations Team.
• Create new software to provide alerts and notifications from HPC system APIs into the monitoring pipeline.
• Builds and maintains application/tool configurations to ensure software runs reliably as data and user demands grow.
• Collaborate with other groups at NERSC to ensure that communication and workflows are clearly understood.
• Work closely with other NERSC groups to coordinate center-wide maintenance activities and manage diagnostic and notification software during maintenance periods.
• Perform regular physical and logical walkthroughs of the data center floor to monitor environmental health, power distribution units, and cooling infrastructure to ensure peak operational efficiency.
• Provide accurate information in the trouble ticketing system for outages, maintenance updates, and other incidents so that workflows and protocols can be appropriately tracked by others.
• Work on and…