AWS Observability or Grafana Architect

AWS Observability Architect

Location:

Warren, NJ (Onsite)

We are seeking a highly skilled AWS Observability Architect with deep, hands‑on expertise in designing and implementing enterprise‑grade observability platforms on AWS – with Grafana as the primary observability tool and Open Telemetry as the instrumentation standard. This is a technical specialist role requiring genuine implementation experience, not platform familiarity.

The ideal candidate has personally architected and delivered large‑scale observability solutions for production AWS environments – building telemetry pipelines, designing dashboards that operations teams actually use, and creating alerting frameworks that reduce MTTR rather than add noise. You understand the full observability stack: from application instrumentation with Open Telemetry SDKs through to Grafana dashboards consumed by SREs, on‑call engineers, and engineering leadership.

This role sits at the intersection of cloud infrastructure, software engineering discipline, and operational excellence – requiring someone who can design an enterprise observability architecture in the morning, write a Grafana dashboard query in the afternoon, and advise a development team on Open Telemetry instrumentation strategy the next day.

• Define and own the enterprise observability architecture for AWS environments – establishing the target‑state design across the four pillars of observability: metrics, logs, traces, and events.
• Design end‑to‑end telemetry pipelines – from instrumentation at the application and infrastructure layer through collection, processing, storage, and visualisation – with Grafana as the enterprise observability platform.
• Develop observability standards and reference architectures – defining how AWS workloads across compute (EC2, EKS, ECS, Lambda), storage, networking, and managed services should be instrumented, collected, and visualised consistently across the organisation.
• Establish signal‑to‑noise discipline across the observability platform – designing alerting frameworks that surface actionable signals, eliminate false positives, and ensure on‑call engineers are alerted only when human intervention is genuinely required.
• Define observability maturity roadmaps for client environments – assessing current‑state coverage, identifying gaps, and building a phased improvement plan from reactive monitoring to proactive, AIOps‑ready observability.
• Drive Fin Ops for observability – optimising telemetry data volumes, retention policies, and Grafana Enterprise licensing costs to ensure the observability platform itself does not become a significant cost centre.

• Architect, deploy, and operate Grafana Enterprise or Grafana SaaS as the primary observability platform – including high‑availability Grafana deployment on AWS (EKS‑based or managed via Grafana Cloud), data source federation, RBAC configuration, and enterprise plugin management.
• Design and implement Grafana data source integrations across the AWS observability ecosystem:
  • Amazon Cloud Watch – metrics, logs, and alarms as a core AWS data source
  • Grafana Mimir – for scalable, long‑term Prometheus‑compatible metrics storage
  • Grafana Loki – for cost‑efficient, label‑based log aggregation at scale
  • Grafana Tempo – for distributed tracing storage and trace‑to‑log‑to‑metric correlation
  • Amazon Managed Service for Prometheus (AMP) – for AWS‑native Prometheus metrics
  • Amazon Open Search – for log analytics and full‑text search use cases
  • Elasticsearch / Open Search – for existing log infrastructure integration
• Build and maintain a Grafana dashboard library – covering infrastructure health, application performance, SLO/SLA tracking, capacity planning, cost visibility, incident response, and executive reporting – using reusable, variable‑driven, and consistently styled templates.
• Implement Grafana alerting at enterprise scale – including alert routing, notification policies, silence management, and integration with Pager Duty, Ops Genie, Service Now, and Slack for multi‑channel incident notification.
• Configure Grafana RBAC and team structures – designing role hierarchies, folder…