Software Engineer, Observability

We have an opening for a Staff Software Engineer on our Infrastructure Team, with specific focus on Observability - both internal and customer-facing
. This is an opportunity to join a mission‑critical engineering team that is driving the productivity and reliability of Temporal's developers and core platforms, respectively. We are a passionate team of talented developers who truly care about our mission and enjoy working deeply across the entire Temporal ecosystem to be a force multiplier across the organization.

• Core observability tech stack:
  Clickhouse, Prometheus, Grafana, Loki, Thanos, Kubernetes, Temporal.
• Our recent observability project had the biggest impact to Temporal margins of any engineering effort in company history while also unlocking unprecedented operator capabilities.
• "Observability is the beating heart of our ability to deliver a mission critical service"
  -Preeti Somal, Executive VP of R&D.

• Lead the end‑to‑end Software Development Lifecycle: goals & requirements solicitation, design & review, implementation, operationalization & deployment, support & maintenance.
• Lead feature design, review with stakeholders, iterate to incorporate feedback and drive consensus.
• Clearly document design choices and operational knowledge to successfully deploy and manage the software you develop.
• Provide appropriate test and production readiness coverage for unit, integration, and performance of your feature ownership area.

• Set a high bar for technical excellence and take pride in the software you develop.
• Design and build multi‑component, distributed systems that operate at scale.
• Investigate issues with a methodical approach to identify a root cause.
• Understand performance and reliability implications of design options e related tradeoffs.
• Participate in the team’s on‑call rotation.

• Expert‑level knowledge of architecture and services of assigned domain. Strong command over all aspects of the Temporal ecosystem. This role provides a strong opportunity for diverse technical exposure!
• Investigate and understand ways to best leverage Temporal's own software to power our mission.
• Deeply understand the needs of Temporal internal developers and external customers, and leverage that knowledge for product development and feature design.

• Lead and mentor fellow team members.
• Participate in design reviews and contribute to design of other features.
• Share design principles for building reliable systems at scale.

• User‑first mindset. You’re excited by the opportunity to empower others through tooling, and enjoy deeply internalizing user goals and use cases to build effective solutions.
• Motivated by impact. You are driven by a desire to make positive things happen.
• Strong opinions about tools and technology that are equally balanced by a pragmatic drive for impact.
• Ability to work in a self‑directed manner in a fast‑paced environment.
• Excellent collaboration and communication skills.

• Demonstrated ability to develop horizontally scalable, resilient, and high performance distributed systems in a production environment.
• Experience designing, implementing, deploying, and supporting large scale, geographically distributed observability and/or high throughput data streaming/processing pipelines, or similar.
• Expert in one or more high‑level programming languages, preferably Go.
• Expert‑level Kubernetes skills.
• Expert‑level query development skills, preferably SQL.
• Hands‑on experience with one or more cloud providers, preferably AWS, or GCP.
• Thorough understanding of computer architecture, operating systems, and networking.
• Familiarity with best practices regarding monitoring, instrumenting, and configuring infrastructure.

Temporal Technologies is…