Lead Engineer - L3 Observability

Presidio, Where Teamwork and Innovation Shape the Future At Presidio, we're at the forefront of a global technology revolution, transforming industries through cutting-edge digital solutions and next-generation AI. We empower businesses - and their internal customers - to achieve more through innovation, automation, and intelligent insights.

The Role Level 3 Managed Service Engineer serves as the senior escalation point and technical authority for end-to-end observability across the environment. This role is responsible for resolving or driving resolution of highly complex incidents spanning application, infrastructure, network, and third-party dependencies, while owning observability strategy, Datadog architecture, monitoring standards, and continuous improvement. The engineer is expected to guide L1 and L2 engineers, lead root cause analysis and major incident reviews, and partner with cross-functional teams to improve detection, reduce noise, and strengthen operational resilience for both 3-tier and cloud-native applications.

Key Responsibilities Serve as the final escalation and technical authority for complex, cross-domain incidents involving application, infrastructure, network, and third-party dependencies.

Own the end-to-end observability architecture and strategy in Datadog, including instrumentation standards, tagging strategy, dashboard governance, and monitor design principles.

Lead root cause analysis for major/critical incidents and drive post-incident reviews, ensuring permanent fixes and preventive monitoring are implemented.

Define and continuously improve monitoring coverage for new and existing services, including 3-tier applications, microservices, and cloud-native workloads.

Design and govern SLOs/SLIs aligned to business-critical services, and advise application and platform teams on instrumentation best practices.

Drive automation initiatives:
Datadog-as-code (Terraform/API-based configuration management), auto-remediation workflows, and integration with CI/CD pipelines.

Partner with application, infrastructure, network, and security teams to close monitoring gaps and reduce mean time to detect (MTTD) and mean time to resolve (MTTR).Own capacity planning and performance engineering insights derived from long-term observability data trends.

Manage Datadog platform health: cost optimization, license/usage governance, data retention, and integration management.

Mentor L1 and L2 engineers, review their RCA quality, and build enablement material (runbooks, training, best practices).Act as a key stakeholder in client governance calls, presenting trends, risks, and improvement roadmaps.

Evaluate and pilot new Datadog features or complementary observability tools, and lead their rollout.

Define escalation matrices and coordinate cross-team response during major outages.

Required Skills and Experience6+ years of experience in Managed Services, SRE, or platform engineering roles, with at least 3 years of hands-on ownership of a Datadog (or equivalent) observability platform ert-level knowledge of Datadog, including:

Platform architecture: organization/team structuring, RBAC, tagging strategy, and multi-environment setups.

APM: distributed tracing at scale, service dependency mapping, and instrumentation strategy across polyglot environments.

Synthetics and RUM: designing comprehensive digital experience monitoring strategies tied to business KPIs.Log Management: pipeline architecture, log-based metrics, and cost/volume governance.

SLOs, error budgets, and anomaly/forecast-based alerting at an advanced level.

Deep understanding of infrastructure and cloud architecture: compute, containers, orchestration (Kubernetes), auto-scaling, and multi-cloud/hybrid environments.

Deep understanding of networking: routing, load balancing, CDN, DNS, and complex connectivity troubleshooting across hybrid environments.

Strong incident and problem management leadership experience, including major incident command and post-incident review facilitation.

Experience defining and governing observability standards across multiple teams or business units.

Application & Architecture Understanding (technology-agnostic)
Expert-level understanding of 3-tier and N-tier application architectures, including how failures cascade across presentation, application, and data layers.

Expert-level understanding of cloud-native and distributed architectures: microservices, event-driven systems, service meshes, API gateways, and asynchronous messaging.

Ability to…