Shelton - Infrastructure - Engr- Observability Sr

Shelton
- Infrastructure
- Engr
- Observability Sr Subway

Compensation $ - $/year

Schedule Full time

Job Type Permanent

Industry It and internet

Description

Sr. Engineer, Observability

We are Subway Headquarters — a dedicated team of professionals supporting thousands of franchisees around the globe. Ready for a fresh, new career? Look no further, because one of the world's most iconic brands can help you get there.

At Subway, "better" is baked into our DNA. We are a brand that believes in continued improvement — in our lives, our businesses, and our planet. From the handshake that started our very first sandwich shop to earning our position as one of the world's leading restaurant brands, we've always embraced change and the path ahead. And today, we're making better living way easier.

Our purpose is about more than the food we serve in our restaurants. It's centered on fueling healthy businesses and healthier lives. It is one of the most exciting times to join the Subway team and contribute to our transformational journey.

We have an exciting opportunity to support our Technology team as a Senior Engineer, Observability, based in Shelton, CT.

This is a senior individual-contributor role. The Senior Engineer, Observability is responsible for implementing and managing observability across Subway's infrastructure and applications — servers, networks, cloud services, databases, restaurant point-of-sale and kiosk estates, mobile apps, and web platforms. The role owns configuration, maintenance, and documentation of our observability platforms, and partners with technology stakeholders through instrumentation planning, service reviews, and operational enablement so that monitoring keeps pace with the business.

Subway's observability estate is large and actively changing: business-event telemetry now runs in production across middleware, order, payment, and restaurant-device flows; a network and security logging domain has recently been onboarded; and incident routing is moving onto Pager Duty. Alongside building coverage, this role carries real responsibility for signal quality and for the cost of the telemetry we collect and query.

If you feel that this is the role for you, and you are successful with your application, be ready to be Bold, Empowered, Accountable, and ready to have Fun in a fast-paced and agile working environment.

• Design, configure, and maintain observability and alerting platforms
  — Dynatrace (primary), Solar Winds, Pager Duty, Fluent Bit / Calyptia log pipelines, AWS Cloud Watch, Azure Monitor, and Splunk. This covers instrumentation, dashboards and notebooks, alerting and anomaly detection, business-event processing and creation, log pipeline configuration and analysis, and synthetic and real-user monitoring.
• Practice detection engineering, not just alert configuration
  — select the right detection pattern for each failure mode, including silence and dead-man's-switch coverage for feeds that can stop, expected-versus-actual and seasonal-baseline comparison, and success-rate and ratio controls for partial degradation. Backtest new alerts against historical incidents before enabling them and prefer one dimensioned alert over many per-geography copies.
• Improve signal quality and reduce noise
  — review and tune alert configurations, thresholds, dwell times, and routing in collaboration with stakeholders; identify and retire duplicate, unowned, and non-actionable alerts; and make sure every production alert has an owner and a documented response.
• Own telemetry cost and consumption governance
  — understand how each observability capability is billed, attribute consumption to owning teams through cost and product tagging,…