Systems Software Engineer, Cluster Control and Automation

About CommonAI

CommonAI CIC is a non-profit organisation built around collaborative engineering and is backed by a mix of UK Government and private funding in order to design, build and deploy innovative AI systems.

Our Scaling Inference Lab (SIL) is developing simulation grounded in real measurements to help the UK invest in promising AI infrastructure. We connect proposed hardware designs to realistic workloads, helping investors and engineers understand their potential, their limitations and the next experiment worth doing.

The Opportunity

Build the software that lets SIL configure and control real AI infrastructure for reproducible experiments. You will turn the capabilities of our experimental clusters into programmatic controls for changes such as workload placement, resource allocation and inference-serving configuration.

The engineering challenge is making those changes dependable: understanding the current state, checking whether an action can be carried out, confirming what actually happened and handling failures. Your software will make it possible to repeat experiments and understand which configuration produced a result.

Working with measurement and simulation engineers and cluster operators, you will establish useful controls for the first experimental studies. Those interfaces will also provide the actions that future design-space optimisation software can call.

What you will do

- Identify the controls available on the selected clusters and expose them through documented, programmatic interfaces with clear supported actions.

- Automate configuration and experiment setup across relevant parts of the stack, such as workload placement, serving settings, resource allocation or device configuration.

- Build reliable execution with precondition checks, confirmation of the resulting state, error handling and recovery to a known configuration where supported.

- Record requested actions, outcomes and resulting configuration so experiments can be reproduced and measurements interpreted correctly.

- Coordinate access, permissions and changes with cluster operators, and test success, failure and recovery paths within an agreed experimental scope.

- Design interfaces that future optimisation software can call, with clear capability limits and explicit outcomes when an action cannot be completed.

An example early project would be to build an interface for changing serving configurations and demonstrate a repeatable experiment across two configurations, including checks of the applied settings and recovery from a failed change.

- Experience building systems software or automation that configures and controls real computing infrastructure.

- Strong programming and debugging skills, with practical understanding of Linux and the relevant hardware/software interfaces.

- Depth in one or more of: cluster or HPC automation, distributed systems, orchestration, device control, networking, or experiment-control software.

- The ability to reason about partial failure, state, permissions and the consequences of changing a running system.

- Practical experience using AI coding agents, with the judgement to validate generated code and test changes that affect hardware or infrastructure.

Experience with accelerators, job-control systems, Kubernetes, Slurm, operating systems (NixOS, CoreOS) or IaC tools (Terraform/Open Tofu) would be useful. Depth in a subset is sufficient; the initial controls and integration choices will follow the needs and capabilities of SIL's clusters.

Why join us

- Help build UK AI capability and improve how investment and compute serve the public good.

- Build a network spanning UK hardware SMEs, major technology companies, venture and public investors, and leading universities.

- Shape a…