AI​/ML Operations & Assurance Engineer

• Active TS/SCI clearance with CI Polygraph.

• BS/BA and 5 years of experience, an additional 4 years of experience may be considered in lieu of a degree.

• Approximately 5–8 years of experience in platform operations, site reliability engineering, cloud engineering, software operations, applied AI/ML operations, or production application support.

• Experience operating and troubleshooting production software applications or cloud services.

• Experience with monitoring, logging, metrics, incident response, and configuration or change management.

• Experience evaluating software behavior through repeatable test cases, regression testing, quality measures, or operational performance analysis.

• Working knowledge of foundation-model or LLM-based applications, including prompts, model endpoints, tool calls, and retrieval-augmented applications.

• Experience with AWS or comparable cloud platforms.

• Strong analytical, troubleshooting, documentation, and communication skills.

• Experience with AWS Bedrock or comparable managed foundation-model services.

• Familiarity with LLM or agent evaluation frameworks, automated evaluation harnesses, or AI application testing.

• Experience with Open Telemetry, Amazon Cloud Watch, Open Search, Prometheus, Grafana, or comparable observability technologies.

• Familiarity with Service Now, IT service management, and incident-management processes.

• Experience with Git Lab, CI/CD pipelines, and controlled configuration promotion.

• Familiarity with Kubernetes and containerized application environments.

• Experience monitoring cloud utilization, model consumption, and service costs.

• Knowledge of risk management frameworks, continuous monitoring, or security evidence collection.

• Experience supporting classified, restricted, or highly regulated environments.

We are seeking an AI/ML Operations & Assurance Engineer to monitor, evaluate, and maintain production AI-enabled applications, foundation-model integrations, and agentic workflows. This role focuses on operational reliability, workflow performance, evaluation and regression testing, configuration control, usage and cost monitoring, and the safe introduction of AI-related changes into production environments.

The ideal candidate combines cloud and application operations experience with an understanding of large language model (LLM) applications, agent behavior, and AI-specific operational risks. The engineer will collaborate with AI/ML developers, site reliability engineers, cybersecurity teams, and platform operations personnel to maintain reliable, auditable, and secure AI-enabled services.

• Monitor availability, latency, error rates, model and token usage, service costs, tool-call failures, and workflow completion metrics for production AI services.

• Integrate AI service health and agent-workflow telemetry into observability dashboards, monitoring systems, and alerting processes.

• Investigate failed, stalled, repetitive, or abnormal AI workflows and coordinate remediation with software, platform, and AI engineering teams.

• Maintain operational runbooks, escalation procedures, troubleshooting guidance, and recovery processes for AI-enabled applications.

• Distinguish infrastructure and application failures from issues associated with model behavior, prompts, retrieval sources, or agent orchestration.

• Develop and maintain repeatable evaluation scenarios for critical AI workflows, developer assistants, and model-service updates.

• Assess task completion, output quality, error patterns, inappropriate tool use, and other indicators of operational suitability.

• Coordinate regression testing before changes to models, prompts, tools,…