Consulting/Principal Site Reliability Engineer
-
IT/Tech
Job Description & How to Apply Below
We're looking for a Consulting/Principal Site Reliability Engineer to guide our observability and reliability initiative for Government Systems, running on Azure Government. This is an advanced individual-contributor role for someone with a proven track record leading an organization through a transition to SLO-based reliability engineering — you've done this before, and you know what it takes to get development teams bought in.
This is a hands-on SRE role, not a software development role: you'll work alongside other SREs on the initiative, guiding development teams on telemetry and instrumentation strategy rather than writing that code yourself. You'll operate as the go-to expert on Datadog and observability practice, helping define meaningful SLOs and error budgets and making data-driven reliability decisions. You'll also help train and mentor junior SREs, working within priorities set by the SRE Manager.
Eligibility:
This role requires U.S. citizenship and the ability to obtain and maintain a Public Trust clearance.
- Guide the observability and reliability initiative, working alongside other SREs to set the technical direction for how the organization measures and improves reliability
- Lead the transition to SLO-based reliability engineering, building on prior experience doing this elsewhere
- Guide development teams on telemetry and instrumentation strategy for their applications, without doing the application development work yourself
- Serve as the primary technical authority on Datadog, from dashboarding to alerting to SLO tracking
- Recommend service level objectives in partnership with Product and Development teams
- Act as a senior escalation point during incidents
- Help improve and evolve the broader SRE framework
- Build disaster recovery plans, including advanced fault injection scenarios
- Advise on SRE training curriculum and content, and help set and follow priorities established by the SRE Manager
You'll Work With Cloud:
Microsoft Azure Government Server Environment:
Linux and Windows CI/CD:
Azure Dev Ops Pipelines, Git Hub Actions Infrastructure as Code:
Terraform Observability:
Datadog (primary) AI-assisted development:
Git Hub Copilot
- Observability (core focus) Significant, hands-on experience with Datadog — dashboards, alerting, SLO/error-budget tracking, and telemetry pipelines
- Proven experience leading an organization's transition to SLO-based reliability engineering, not just supporting one
- Track record guiding development teams on instrumentation and telemetry strategy (advisory, not hands-on application coding)
- Deep technical understanding of observability techniques across the full stack; able to bring clarity to complex incidents or performance issues
- Able to build templated observability dashboards and configuration as code so others can implement quickly
- Able to set and influence appropriate SLOs and Error Budgets org-wide
- Comfortable operating across both Linux and Windows server environments
- Incident Management Experience preparing and assessing SREs for on-call readiness, including mentoring and identifying training needs
- Skilled at running post-mortems where participants feel safe to contribute; fosters a culture of learning from failure
- Comfortable acting as a senior on-call escalation point and guiding restoration efforts
- Experience partnering with SRE and Development leads to identify and fix root causes of excessive alerting and pager load
- Design for Reliability Advanced understanding of systems design, including high availability, deployment, and recovery techniques, plus failure modes and system behaviors
- Specialized depth in one or more areas such as denial-of-service protection or containerization, with the ability to…
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction