Observability Engineer​/Production Engineer

Overview

The Observability Engineer / Production Engineer will design and optimize complex queries over large telemetry datasets to support AI‑driven incident investigation and system reliability. This role involves creating synthetic datasets simulating cloud infrastructure logs and working extensively with tools like Clickhouse and Grafana. Candidates should have 3+ years of experience in observability or production engineering with strong SQL and query optimization skills.

Sepal AI builds the world’s hardest tests for AI grounded in real‑world software systems. We are looking for an Observability Engineer with 3+ years of experience to help us understand, debug, and operate complex production systems  will work deeply with production logs, metrics, and traces, building tasks and datasets that can teach AI to investigate incidents, analyze performance, and improve system reliability.

• Design complex, distributed queries over massive log and telemetry datasets.
• Explore creative ways to challenge AI's reasoning ability and log analysis skills.
• Create and manage synthetic datasets that simulate real‑world Dev Ops, observability, or cloud infrastructure logs.

• 3+ years of experience in observability engineer, production engineer, or platform engineer roles.
• Deep expertise in Clickhouse and Grafana, with a focus on large‑scale query optimization, schema design, and performance tuning.
• Strong SQL skills: you know how to reason through performance problems and spot inefficient query patterns.

Pay: $50 – 85/hr depending on experience. Remote, flexible hours. Project timeline: 5–6 weeks.

Observability Engineer, Production Engineer, Clickhouse, Grafana, SQL optimization, Distributed systems, Telemetry data, Cloud infrastructure logs, AI debugging, System reliability.