Site Reliability Engineer; AI Observability Job Austin area,Texas USA,IT/Tech

Position: Site Reliability Engineer (AI Observability)

Appnovation is a global, full-service digital partner that combines Strategy, Experience & Design, Engineering and Managed Services. We build digital solutions that deliver real impact today and serve as foundations for future growth. Bold ambition. Practical action. Endless possibilities
We’re looking for a Site Reliability Engineer to keep a shared observability platform for LLM-based applications running for a global life sciences client. The platform is built on Langfuse and self-hosted on Kubernetes on AWS, with Click House as the analytical store, PostgreSQL for metadata and Redis as the ingestion queue, all delivered through Argo CD
Two things need building rather than maintaining. The platform has no monitoring, alerting or defined service levels today, and you will own putting them in place. Infrastructure is defined as code throughout, in Kubernetes manifests, Helm values and Argo CD applications
Alongside the platform itself, you will own the runbooks that make an incident survivable by someone other than the author, and the onboarding and support path for the internal teams that depend on the platform
Platform Operations:
Diagnose and resolve failures across Click House, PostgreSQL, Redis, the ingestion workers and the Kubernetes layer beneath them, including ingest back pressure from queue depth and worker drain behaviour
Click House Operations:
Own Click House under the operator model, including Keeper quorum, replication, shard and replica topology, and S3 storage tiering
Provable Backup and Restore:
Rehearse the restore, time it, document it and test it against its failure modes, rather than assuming a successful backup job means a recoverable system
Safe Upgrades:
Plan and rehearse upgrades in a lower environment, with a rollback plan that works even if a migration is only partly complete
Monitoring and Service Levels:
Build monitoring, alerting and service levels from scratch, so problems are found here before a user reports them
Infrastructure as Code:
Maintain Kubernetes manifests, Helm values and Argo CD applications so every change goes through the delivery pipeline
Runbooks and SOPs:
Write and maintain runbooks and SOPs that let a colleague resolve an incident without the author present
Onboarding and Support:
Run the onboarding and support path for internal teams that depend on the platform, and triage what they bring
Automation:
Turn recurring operational work into automation
Upgrade Partnership:
Work with the platform engineer who owns what the platform offers. They decide what to adopt and how it is configured; you own the migration and its rollback

PostgreSQL and Redis operations deep enough to debug metadata-store and queue problems, including back pressure and worker drain
Hands‑on experience running Click House in production, including replication and Keeper quorum, shard and replica topology, and backup and restore, ideally run through an operator
Hands‑on experience with Kubernetes on AWS (managed EKS), with routine work done through Helm values and Argo CD applications
Experience building monitoring and alerting from scratch, including service levels that reflect what users actually experience rather than what is easy to measure
Strong operational writing: runbooks, SOPs and post‑incident reviews that a colleague can follow unaided during an incident
Experience upgrading self‑hosted software safely, including schema migrations, rehearsal in a lower environment and a rollback plan for a partly completed migration
You don’t trust a backup until you’ve restored from it
You automate anything you have to do twice
Prior experience and connections in the Life Sciences industry is preferred
You stay calm in incidents and focus on fixing the process afterward
You have prior experience in consulting
You write runbooks for the person on call at 3am, not for yourself
You work well inside a client team and build trust quickly
You want to find problems before users do
Observability engineering, including Open Telemetry Collector pipelines, alerting design and Grafana dashboards
OIDC or enterprise SSO integration with a corporate identity provider
Git Hub Actions for plan and apply pipelines with approval gates
Experience running LLM observability tools such as Langfuse, Lang Smith or Arize Phoenix
Experience in pharma, life sciences or another regulated industry