Senior Scalability Engineer - Observability

Position Summary

Senior Scalability Engineer focused on observability platform development and engineering productivity. Define, own, and build Judi Health's organization‑wide observability strategy, tooling, and platform products. Architect and develop a custom observability platform that gives engineering teams powerful, fast, and cost‑effective visibility into every layer of our infrastructure—from application logs and metrics to distributed traces. Build production‑grade internal products using React/Type Script frontends with Python and Rust backends, creating tools that fundamentally improve how engineers debug, monitor, and optimize their systems.

• Architect observability platform:
  Design, implement, and maintain the LGTM stack (Loki, Grafana, Tempo, Mimir/Prometheus) as the primary observability platform.
• Build internal observability products:
  Design and develop production‑grade platform products with React/Type Script frontends and Python/Rust backends for log search, metrics visualization, and trace analysis.
• Develop custom log indexing systems:
  Build high‑performance log indexing solutions using Rust that provide sub‑second search across billions of log lines.
• Integrate SQL analytics for logs:
  Implement solutions leveraging AWS Athena or similar SQL query engines for ad‑hoc log analysis and historical queries.
• Create advanced query interfaces:
  Build web interfaces that allow engineers to query logs, metrics, and traces with saved queries, templates, correlation analysis, and pattern detection.
• Balance cloud‑native and open‑source:
  Architect solutions that leverage both AWS‑managed services and open‑source tooling to optimize cost, performance, and operational flexibility.
• Integrate AWS observability:
  Provide unified visibility across managed and self‑hosted infrastructure.
• Build intelligent alerting:
  Develop dashboards, monitors, and alerting systems that reduce noise, detect anomalies, and help teams respond quickly.
• Partner with engineering teams:
  Integrate observability into services, establish logging and metrics standards, and instrument code effectively.
• Enable performance optimization:
  Provide foundation that allows identification of bottlenecks and measurement of platform stability.
• Establish observability standards:
  Define and document standards for structured logging, metric naming, trace instrumentation, dashboard design, and query best practices.
• Drive platform adoption:
  Lead workshops, create documentation, and build self‑service tooling to democratize observability across engineering.
• Demonstrate technical leadership:
  Mentor engineers, lead architecture reviews, and represent the Scalability team in cross‑functional planning.
• Work in an Agile/Scrum environment to continually deliver value.
• Code of Conduct:
  Adhere to the Capital Rx Code of Conduct.

• 10+ years of software or infrastructure engineering with progression into technical leadership.
• Several years of leading technical initiatives, building platform products, or serving as a subject‑matter expert on observability infrastructure.
• Strong experience with React/Type Script and Python (Flask/SQL Alchemy) for frontend and backend services.
• Deep production experience with the LGTM stack:
  Loki, Grafana, Tempo, Prometheus/Mimir.
• Extensive experience with AWS Cloud Watch Logs and Metrics, including custom metrics, log insights, and dashboard creation.
• Production experience with SQL‑based log analytics using AWS Athena, DuckDB, or similar engines.
• Demonstrated ability to architect solutions leveraging both managed cloud services and open‑source tooling.
• Hands‑on experience building or operating search systems using Open Search, Elasticsearch, Lucene, or Tantivy.
• Experience building high‑performance systems that process large volumes of data efficiently.
• Deep understanding of distributed systems and microservices architectures.
• Proven record handling high‑volume structured and unstructured logging data, identifying patterns, and building efficient search/query solutions.
• Product mindset:
  Build internal platform products that engineers love, with attention to UX, performance, and reliability.