Skip to content
×

Research Scientist​/Engineer – Training Infrastructure

Listing for: lumaai
Full Time, Apprenticeship/Internship position Listed on 2026-11-15
Job specializations:
Salary/Wage Range or Industry Benchmark: 180000 - 240000 USD Yearly USD 180000.00 240000.00 YEAR

Job Description & How to Apply Below

Position: Research Scientist / Engineer – Training Infrastructure
Role overview

This position sits at the intersection of research and infrastructure engineering, focused on building the distributed training systems that power large-scale multimodal foundation models running across thousands of GPUs. The work spans hard PyTorch and CUDA engineering alongside distributed-systems design, with an emphasis on advanced parallelism, training stability, and high utilization across massive clusters. It is built for engineers who have already solved real foundation-model training problems rather than those still ramping into distributed systems.

Responsibilities
  • Design, implement, and optimize distributed training systems that scale to thousands of GPUs while preserving throughput and stability
  • Research and integrate advanced parallelization strategies, including FSDP, tensor, pipeline, and expert parallelism
  • Build monitoring, visualization, and debugging tooling that makes large training runs observable and diagnosable end to end
  • Tune training stability, convergence behavior, and resource utilization across massive GPU fleets
First 90 days
  • Days 1–30 — Immerse and diagnose the current training stack, identifying where stability and utilization break down at scale
  • Days 30–60 — Ship a parallelization or stability improvement that measurably helps a real production training run
  • Days 60–90 — Build the monitoring and reliability tooling that keeps multi-thousand-GPU runs healthy and efficient
Requirements
  • Extensive hands-on experience with distributed PyTorch training and the parallelisms used in foundation-model development
  • Deep understanding of GPU cluster architecture, including networking and storage subsystems
  • Familiarity with communication libraries such as NCCL and MPI, paired with a track record of distributed-system optimization
Nice to have
  • Strong Linux systems administration and scripting skills
  • Experience managing training runs across 100+ GPU deployments
  • Background with containerization, orchestration, and cloud infrastructure
To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction

Search for further Jobs Here:

SEARCH JOBS: 10+ Million Listings
(Try combinations for better Results! Or enter less keywords for broader Results)
Education Level
Experience Level (years)
Posted in last:
Salary