Research Scientist, Distributed Workflows

Research Scientist, Distributed Workflows

Oak Ridge National Laboratory (ORNL) is seeking a Research Scientist to advance distributed workflow orchestration and cross-facility integration in support of the Genesis Mission, the American Science Cloud, and the Oak Ridge Leadership Computing Facility (OLCF). Working within the Workflows and Ecosystem Services (WES) Group at the National Center for Computational Sciences (NCCS), you will design and deploy APIs, service meshes, and workflow orchestration systems that bridge leadership computing resources, instruments, and cloud platforms.

You will collaborate with scientists, facility operators, and DOE laboratory partners to lower barriers to scalable, automated scientific computing.

Major Duties/Responsibilities:

• Distributed Workflow Orchestration:
  Architect, implement, and operate cross-facility workflow orchestration systems that integrate job schedulers (e.g., SLURM), data movement services (e.g., Globus), and computing endpoints across multiple DOE facilities.
• API and Service Mesh Development:
  Design and implement RESTful APIs and service meshes that expose OLCF computing resources to diverse scientific workflows and external facilities.
• Genesis Mission / American Science Cloud Integration:
  Lead ORNL's technical engagement with the Genesis Mission, architecting cross-facility workflow and data infrastructure that connects leadership computing, experimental facilities, and cloud platforms in support of DOE's vision for integrated science.
• Agentic Workflow Design and Autonomous Control Systems:
  Design and implement agentic workflow architectures that enable autonomous, closed-loop scientific experimentation across HPC and instrument facilities.
• Scientific Domain Support:
  Collaborate with domain scientists in areas such as climate science, materials science, and autonomous experimentation to translate research objectives into scalable, automated workflow solutions.
• Scripting and Tooling:
  Develop command-line tools and automation in Python, Bash, and/or C/C++ to encapsulate workflow steps, manage configuration files (e.g., YAML/JSON), and implement robust logging, error handling, and checkpoint/retry strategies.
• Operational Reliability and Optimization:
  Diagnose job failures, mitigate bottlenecks, and improve throughput, latency, and resource utilization. Use scheduler and Linux tools (e.g., sacct, squeue, coreutils, ssh, tmux, top, iostat) to monitor, analyze, and tune workflows.
• Research and Publication:
  Conduct original research on workflow systems, orchestration, and data management. Publish findings in high-impact, peer-reviewed venues and present work at major conferences (e.g., SC, HPDC, PEARC, eScience).

Basic Qualifications:

• D. in Computer Science, Computer Engineering, or a closely related field.
• At least 2 years of experience designing or operating distributed systems, APIs, or workflow orchestration frameworks.
• At least 1 year of programming experience in Python, with working familiarity in Bash and/or a compiled language (C/C++ or similar).
• Experience with Linux-based HPC environments and familiarity with job scheduling systems (e.g., SLURM or PBS).
• Strong verbal and written communication skills, with the ability to collaborate across technical and scientific teams.

Preferred Qualifications:

• Demonstrated experience publishing research in high-impact venues such as SC, IPDPS, HPDC, PEARC, or IEEE eScience.
• Familiarity with RESTful API design, containerization (Docker/Singularity/Apptainer), and microservice architectures.
• Experience working in or coordinating across DOE national laboratory environments, particularly in the context of integrated or cross-facility research infrastructure.
• Background in scientific data management, provenance tracking, or metadata systems.
• Demonstrated ability to lead technical projects, manage milestones, and coordinate multidisciplinary teams.
• Familiarity with basic cyber-security principles (e.g., SSH key hygiene, least privilege, network segmentation) as applied to workflow and API design.

Special Requirements:

Remote/Hybrid Eligibility:
This position is eligible for hybrid schedule within approved U.S. States, subject…