Principal Systems Operations Engineer; SysOps

Lead secure, reliable production infrastructure for AI software at Staff or Principal level, with hands‑on ownership of deployment, observability, resilience, and recovery.

Build the operational foundation for AI software used in regulated industries and critical operations. Partner with software engineering, data science, and security colleagues to turn product requirements into infrastructure that teams can deploy, understand, protect, and recover.

This is a hands‑on engineering role. Automate repeatable operations, make failure visible, and reduce the need for manual intervention. Treat access boundaries, customer information, change control, and operational evidence as part of the system design.

Staff Systems Operations Engineers lead the reliability and operational design of substantial services or platforms, establish practical deployment and recovery standards, and mentor colleagues. Principal Systems Operations Engineers set infrastructure strategy across products, resolve risks that span systems and teams, and guide long‑term architecture, capacity, and operational investment.

Staff level requires at least 8 years of relevant professional experience;
Principal level requires at least 12 years. Demonstrated technical leadership, scope, judgment, and impact determine the level under consideration.

• Design, provision, and operate cloud infrastructure, Linux systems, networking, storage, databases, and containerized services for AI products.
• Build reviewed, version‑controlled infrastructure as code and configuration automation; make environment changes reproducible and traceable.
• Develop CI/CD pipelines with appropriate testing, artifact integrity checks, staged rollout, rollback, and controlled access to production.
• Define service‑level objectives with product and engineering teams; build metrics, logs, traces, dashboards, and actionable alerts around user‑visible behavior.
• Lead incident investigation and recovery, maintain useful runbooks, and turn incident reviews into verified improvements to systems and operating practices.
• Implement least‑privilege access, network isolation, secrets management, encryption, patching, and vulnerability remediation in partnership with security colleagues.
• Design backup, restore, disaster recovery, and service failover procedures; regularly exercise them against agreed recovery objectives.
• Operate model‑serving and AI workflow dependencies with attention to resource limits, request queues, timeouts, rate limits, retries, and external‑provider failures.
• Plan capacity and investigate performance and infrastructure cost; identify bottlenecks before they affect critical workflows.
• Maintain operational documentation and evidence of access, configuration, and production changes to support customer and assurance reviews.
• Improve developer self‑service through reliable environments and automation; mentor engineers in production readiness and operational ownership.

• Staff level: at least 8 years of relevant professional experience in systems operations, site reliability engineering, platform engineering, Dev Ops, or production infrastructure.
• Principal level: at least 12 years of relevant professional experience in systems operations, site reliability engineering, platform engineering, Dev Ops, or production infrastructure.
• Prior professional experience working at an AI‑forward company is required.
• A record of operating consequential production services and leading architecture, reliability, or operational improvements across teams.
• Deep hands‑on knowledge of Linux, networking, DNS, TLS, identity and access management, storage, and distributed‑system failure modes.
• Strong infrastructure automation skills using Python or Go and…