Senior Reliability Engineer Job Los Angeles California USA,Software Development

We’re looking for a Senior Reliability Engineer to make Hims & Hers systems measurably more reliable: building the observability, tooling, and automation that catch problems before people do
Recent work at this level includes preparing our stack for 32x baseline traffic during high-stakes seasonal surges with zero customer-facing issues and improved P95 latency, turning databases into a paved platform with consistent observability and guardrails, and building AI agents that pull Fire Hydrant, Datadog, Jira, and Confluence into a single incident picture
We use AI first: every engineer gets a Claude Enterprise license, and we expect you to use it as a core part of how you investigate, build, and write, not as an afterthought
Own reliability for Tier 1 customer journeys:
Define and instrument SLOs, golden signals, and business-level monitors for the journeys that matter most (checkout, telehealth visits, prescription fulfillment), so that degradation is detected by systems rather than by customers or support tickets. Move our “detected by monitors vs. detected by humans” ratio in the right direction and be able to show it
Engineer for peak load and failure:
Lead capacity and resilience work for high-stakes events and steady-state growth: load-test design, second-by-second analysis of prior events, database and backend bottleneck investigation, and hardening across caching, GraphQL, VPC capacity, and vendor rate limits. Validate before game day, not during it
Build incident response as software:
Mature Fire Hydrant, Datadog, and Jira into one connected pipeline: automated incident and RCA ticket creation, SLO burn-rate and composite alerting with deduplication, Tier 1 alert routing, and enforced post-mortem action tracking. Author and maintain the runbooks and severity standards that make any responder effective on any service
Automate operational excellence with AI:
Build and operate agents and tooling that reduce manual OE work: OER report generation, RCA drafting, stale action-item detection, monitor and runbook gap detection, and Open Claw agents wired to Datadog and Fire Hydrant for first-pass incident triage. Ship these as reusable capabilities, not personal scripts
Be the deep-debugging expert, and make teams better at it:
Teams own debugging their own services, but you are the person they pull in when a problem crosses boundaries: silent service-to-service failures, anomalous traffic, or regressions that span frontend, API, mesh, and database layers
Bring that depth to the hardest cases yourself, then turn what you learn into runbooks, tooling, and pairing so teams can catch and resolve the next one on their own. Partner with Security and product teams on anomaly detection and response, weighing engineering cost and user impact alongside the benefit of any control
Maintain the tooling that informs Operational Excellence reviews:
Produce the tooling that powers the metrics and narrative that go into bi-weekly VP-level OE reviews and the monthly cross-engineering OER, and drive the resulting action items to closure
Raise the bar for others:
Document what you build, onboard teammates to roll it out, and coach engineers across squads on SLOs, blameless post-mortems, and on-call practice

Benefits

Generous PTO:
Take the time you need, when you need it - including generous parental leave
Full healthcare:
High-coverage medical, dental & vision coverage for individuals and families
Retirement planning:
Take advantage of our 401(k) plan including contribution matching
Work from anywhere:
We are a remote-first company, so you can work from anywhere you like in the uS
Robust compensation:
We offer competitive salary bands and stock options
Employee discount:
Employees can take advantage of product…