Research Scientist/Engineer – Training Infrastructure
Job in
Charlotte, Mecklenburg County, North Carolina, 28201, USA
Listing for:
lumaai
Full Time, Apprenticeship/Internship
position
Listed on 2026-11-15
Job specializations:
-
Software Development
Job Description & How to Apply Below
Role overview
This position sits at the intersection of research and infrastructure engineering, focused on building the distributed training systems that power large-scale multimodal foundation models running across thousands of GPUs. The work spans hard PyTorch and CUDA engineering alongside distributed-systems design, with an emphasis on advanced parallelism, training stability, and high utilization across massive clusters. It is built for engineers who have already solved real foundation-model training problems rather than those still ramping into distributed systems.
Responsibilities- Design, implement, and optimize distributed training systems that scale to thousands of GPUs while preserving throughput and stability
- Research and integrate advanced parallelization strategies, including FSDP, tensor, pipeline, and expert parallelism
- Build monitoring, visualization, and debugging tooling that makes large training runs observable and diagnosable end to end
- Tune training stability, convergence behavior, and resource utilization across massive GPU fleets
- Days 1–30 — Immerse and diagnose the current training stack, identifying where stability and utilization break down at scale
- Days 30–60 — Ship a parallelization or stability improvement that measurably helps a real production training run
- Days 60–90 — Build the monitoring and reliability tooling that keeps multi-thousand-GPU runs healthy and efficient
- Extensive hands-on experience with distributed PyTorch training and the parallelisms used in foundation-model development
- Deep understanding of GPU cluster architecture, including networking and storage subsystems
- Familiarity with communication libraries such as NCCL and MPI, paired with a track record of distributed-system optimization
- Strong Linux systems administration and scripting skills
- Experience managing training runs across 100+ GPU deployments
- Background with containerization, orchestration, and cloud infrastructure
To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction