Software Engineer, Observability​/Site Reliability

Psi Quantum is building fault-tolerant quantum computers using silicon photonics. The Observability team builds and operates the telemetry platform that monitors quantum computing systems, on-premises Kubernetes clusters, cloud infrastructure, and custom hardware including control electronics and cryostats. This Staff-level role combines hands-on observability engineering with technical leadership, owning major parts of the stack from design through operation.

• Design, deploy, and operate observability platform across on-premises Kubernetes and cloud, covering collection, storage, querying, visualization, and alerting for metrics, logs, and traces
• Roll out Open Telemetry across the fleet: deploy and configure Collectors, design telemetry pipelines, and help service teams adopt OTel SDKs in Python, Rust, and other languages
• Run Prometheus in production including service discovery, scrape configuration, recording and alerting rules, Alert manager routing, retention, high availability, and long-term storage
• Work with hardware teams to instrument custom devices, control electronics, and cryostats; build exporters and collectors for device telemetry
• Define and publish telemetry standards for metric naming, cardinality budgets, structured logging, and trace context propagation
• Participate in on-call rotation, respond to incidents, diagnose production issues, and lead blameless post-incident reviews
• Mentor engineers, lead design reviews, and coordinate observability needs across software, hardware, research, and operations teams
• Use AI and LLM tools responsibly to accelerate engineering work including code generation, query building, and incident analysis

• 8+ years of professional software, infrastructure, or site reliability engineering experience, including significant time building and operating observability systems in production
• Bachelor's or Master's degree in Computer Science, Computer Engineering, Electrical Engineering, Mathematics, Physics, or closely related technical discipline
• Deep hands-on experience with Prometheus and Grafana, including PromQL, recording and alerting rules, Alert manager, exporters, and dashboard design at scale
• Production experience with Open Telemetry, including Collector, OTLP, SDK, automatic instrumentation, and semantic conventions
• Hands-on experience running workloads on self-managed on-premises Kubernetes and cloud with containers, Helm, and infrastructure as code such as Terraform
• Strong programming skills in Python and at least one of Go or Rust, with ability to write production-quality exporters, Collector components, and automation
• Solid knowledge of Linux and networking fundamentals including processes, file systems, resource limits, DNS, TCP/IP, and TLS
• Experience carrying a pager for production systems, leading incident response, running post-incident reviews, and designing actionable alerts
• Staff-level technical leadership: leading design work, setting standards, mentoring engineers, influencing across teams, and communicating tradeoffs
• Proficiency with modern AI/LLM engineering tools such as coding copilots, model APIs, and retrieval workflows, with disciplined approach to security and code review

• Experience with scalable metrics, log, and trace backends such as Thanos, Mimir, Victoria Metrics, Loki, Tempo, Open Search, or Click House
• Experience collecting telemetry from laboratory or industrial equipment over protocols such as Modbus, OPC UA, SNMP, SCPI, or serial interfaces
• Familiarity with cryogenic, vacuum, laser, or other lab systems, or with software used in laboratories, data centers, or advanced manufacturing
• Experience with high-frequency or high-cardinality time-series data, including downsampling and data lifecycle management
• Experience running observability across multiple sites with hybrid on-premises and cloud environments including constrained network connectivity
• Experience with eBPF-based observability, continuous profiling, or Kubernetes operators
• Experience with SLO tooling, synthetic monitoring, or incident management platforms such as Pager Duty
• Familiarity with Kafka, AWS, or VAST Data
• Contributions to Open Telemetry, Prometheus, Grafana, or related open-source projects
• Background in physics or quantum computing is helpful but not required