Senior Storage & Data Engineer

The Swiss National Supercomputing Centre (CSCS) develops and operates a high-performance computing and data research infrastructure that supports world‑class science in Switzerland. Its user laboratory is available to domestic and international researchers in academia, industry, and the business sector. The centre is operated by ETH Zurich and has offices at its data centre in Lugano and in Zurich.

For this position the work location is either Lugano or Zürich. The contract is for two years.

Storing petabytes is the easy part. The hard part is everything between the moment data lands on disk and the moment a researcher – or a training job – can actually trust it, find it, and use it. Our parallel file systems and object stores already move data fast. What they don't do on their own is tell a scientist where a dataset came from, which transformations produced it, whether it's the version that backed last quarter's published result, or how to feed it to a Data Loader without saturating the I/O subsystem.

That gap – between raw bytes and usable, traceable, reproducible data – is where this role lives. You'll work at both ends: the storage layer (throughput, integrity, tiering at multi‑petabyte scale) and the data layer above it (lineage, provenance, discoverability, access patterns). If you've ever been annoyed that “the data is on the cluster” gets treated as the end of the job rather than the start of it, read on.

• Bridge ingestion and use. Design the pipelines and metadata that turn ingested data into something findable and consumable – catalogs, schemas, and access layers that match how training jobs and simulations actually read, not just where bytes sit.
• Make data traceable. Build lineage and provenance so any dataset, checkpoint, or result can be traced back to its inputs and transformations. Reproducibility is a first‑class requirement here, not a retrofit.
• Tune for the workload. Optimise parallel file systems (Lustre, GPFS) and object storage for the concurrency, small‑file, and large‑checkpoint patterns of distributed GPU training and HPC simulation.
• Operate at scale, safely. Design and run multi‑petabyte storage with the integrity and availability scientific work depends on – erasure coding, redundancy, hot‑to‑archival tiering.
• Automate everything. Deploy and scale storage and data services as code. Snowflake infrastructure doesn't survive at this scale.
• Make it observable. Instrument storage health, capacity trends, and pipeline performance so problems surface before users feel them.
• Translate. Turn real access patterns from domain scientists and ML engineers into technical requirements – and push back when a request would quietly break something downstream.

For a project in the weather and climate domain, aimed at understanding and mitigating the impact of climate change, an opening for two years is available. The initial two‑year contract could potentially be extended or even become permanent.

• A technical degree (CS, engineering) or equivalent experience that demonstrates the same depth.
• Solid storage grounding: file systems (block and object), performance tuning, redundancy (RAID, erasure coding).
• Python, and comfort automating infrastructure (Ansible, Terraform, or similar).
• A working understanding of how ML and scientific workloads consume data – billions of small files, large checkpoints, sharding – and why naive layouts fall over.
• A point of view on data lineage, provenance, or reproducibility – and ideally tooling you've used to enforce it.

• Hands‑on parallel file systems (Lustre, Spectrum Scale/GPFS) or distributed storage (Ceph, VAST).
• Scientific data formats – HDF5, Zarr, Parquet – and opinions on when each earns its place.
• Object storage (S3) interfaced with ML frameworks (PyTorch, Tensor Flow).
• Orchestration (Kubernetes, Argo) and data‑movement tooling.
• Data versioning / cataloguing (e.g. DVC, lake
  
  FS, a metadata catalog) and familiarity with FAIR data principles.
• CI/CD and provisioning:
  Git Lab CI, Hashi Corp Vault, MAAS.

• Hardware and scale you won't find in enterprise IT – and problems with no vendor playbook.
• Work that…