Observability Engineer

Position Overview The Observability Engineer builds and improves the capabilities used to understand the health, performance, and reliability of Goosehead applications and services. This hands-on role focuses primarily on application and service observability and partners with Cloud & Infrastructure on underlying infrastructure telemetry.

The engineer works with Software Engineering, Dev Ops, NOC, Incident Management, and Production Support to instrument services, improve telemetry, establish service-health and alerting standards, and accelerate production diagnosis. New Relic is a key platform, but the role is broader than administration of any single tool.

Core outcome:
Make production health visible, alerts actionable, and failures easier to diagnose.

Key Responsibilities
· Design, implement, and improve observability for production applications, APIs, integrations, and critical services using Open Telemetry for instrumenting metrics, logs, traces, and custom telemetry.
· Partner with Engineering to instrument applications using Open Telemetry for structured logging, distributed tracing, and custom metrics.
· Create and maintain service-health dashboards, alerts, service maps, and diagnostic views that make availability, latency, errors, dependencies, and customer impact clear.
· Establish practical standards for instrumentation, monitoring coverage, alerting, dashboards, Service Level Indicators (SLIs), and Service Level Objectives (SLOs).
· Improve alert quality by reducing noise, false positives, duplicates, and non-actionable alerts; ensure alerts support effective NOC triage and escalation.
· Support Incident Management and Production Support during significant events by correlating telemetry, deployments, dependencies, and system behavior to speed diagnosis.
· Partner with Problem Management to identify observability gaps after recurring or significant incidents and implement preventive improvements.
· Partner with Dev Ops and Cloud & Infrastructure to connect application observability with CI/CD, deployments, cloud, infrastructure, and platform telemetry.
· Build scripts and automation using Power Shell, Python, Bash, APIs, or similar technologies to improve telemetry, configuration, alerting, and operational workflows.
· Help integrate observability and alert/event data with ITSM and incident workflows, including future Service Now capabilities.
· Maintain concise technical standards, reusable patterns, and operational guidance for Engineering and Technology Operations teams.
· Maintain service-based error budgets which our development teams use to dictate feature work.

Qualifications
· Minimum 3 years of hands-on experience in Observability Engineering, SRE, Dev Ops, Application Performance Monitoring, Production Engineering, Application Support Engineering, or a closely related role; 4-5 years preferred.
· Hands-on experience with application performance monitoring, metrics, centralized logging, distributed tracing, dashboards, alerting, and production troubleshooting.
· Experience instrumenting applications or services using Open Telemetry for custom metrics, structured logs, and distributed tracing.
· Strong understanding of APIs, integrations, distributed applications, service dependencies, and common performance and reliability issues.
· Ability to correlate metrics, logs, traces, and change/deployment information to troubleshoot production issues.
· Hands-on scripting or automation experience using Power Shell, Python, Bash, APIs, or comparable technologies.
· Working knowledge of SLIs, SLOs, availability, latency, error rates, alert design, and service-health concepts.
· Working knowledge of cloud environments and the relationship between application and infrastructure telemetry.
·…