Tech Lead – Network Observability

Clockwork.io – Software Driven Fabrics to increase GPU cluster utilization

Clockwork Systems was founded by Stanford researchers and veteran systems engineers who share a vision for redefining the foundations of distributed computing. As AI workloads grow increasingly complex, traditional infrastructure struggles to meet the demands of performance, reliability, and precise coordination. Clockwork is pioneering a software-driven approach to AI fabrics by delivering cross-stack observability to catch and quickly resolve problems,
workload fault tolerance to keep jobs running through failures, and performance acceleration that dynamically routes and paces traffic to avoid congestion.

We are seeking an experienced Tech Lead to lead the architecture, development, and scaling of a high‑performance network monitoring and observability platform. This role will focus on building systems that provide deep visibility into RDMA, RoCE, Infini Band, and TCP/IP networks. The ideal candidate has strong experience in distributed systems, Linux networking, and modern observability stacks (e.g., Grafana/Prometheus).

• Lead architecture, design, and development of scalable network monitoring platforms for high-performance RDMA, RoCE, Infini Band, and TCP/IP infrastructure.
• Build backend telemetry services, observability dashboards, alerts, diagnostics, anomaly detection, SLA monitoring, and traffic analysis workflows.
• Troubleshoot complex production issues across application, OS, server, RDMA, and network layers while optimizing low‑latency collection, aggregation, and alerting.
• Establish engineering standards, drive automation, define technical roadmaps with cross‑functional teams, and mentor engineers on distributed systems and high‑performance networking best practices.

• Bachelor’s or Master’s degree in Computer Science, Computer Engineering, Electrical Engineering, or a related technical field.
• Strong hands‑on programming experience in C++, Go, Python, Rust, or similar systems programming languages.
• Proven experience leading engineering teams, major technical initiatives, or complex infrastructure projects.
• Experience building distributed systems, backend services, telemetry pipelines, or observability platforms.
• Hands‑on experience with RDMA, RoCE, Infini Band, or other high‑performance network fabrics.
• Familiarity with libibverbs, RDMA verbs, RDMA CM, queue pairs, completion queues, memory registration, and related RDMA concepts.
• Strong knowledge of Linux networking, TCP/IP, DNS, HTTP, routing, MTU, congestion control, packet loss, latency, and performance tuning.
• Experience with trace route‑style diagnostics, path discovery, network reachability checks, synthetic probes, or active network measurements.
• Experience with monitoring and visualization platforms such as Prometheus, Grafana, Datadog, Splunk, Open Telemetry, or similar tools.
• Strong debugging skills across software, operating system, server, and network layers.
• Experience operating production systems in Linux‑based environments.
• Strong architectural judgment and ability to design systems for reliability, scalability, and operational simplicity.

• Experience supporting AI/ML, HPC, storage, or GPU cluster infrastructure workloads.
• Experience with large‑scale RoCE or Infini Band deployments.
• Experience with NCCL, distributed training infrastructure, or AI cluster diagnostics.
• Experience with eBPF, XDP, DPDK, perf, tcpdump, Wireshark, ethtool, iproute2, rdma-core, or Linux kernel networking tools.
• Experience with cloud infrastructure on AWS, GCP, or Azure.
• Experience with Kubernetes, service discovery, configuration management, and infrastructure automation.
• Knowledge of security, compliance, and infrastructure best practices.
• Experience designing time‑series data systems, alerting pipelines, or high‑cardinality telemetry platforms.
• A friendly and inclusive workplace culture.
• Competitive compensation.
• A great benefits package.
• Catered lunch.

Compensation for this position will vary based on the skills and experience you bring, as well as internal equity considerations. For candidates hired at the posted level, the expected base…