AI​/ML Operations & Assurance Engineer

We are seeking an AI/ML Operations & Assurance Engineer to monitor, evaluate, and maintain production AI-enabled applications, foundation-model integrations, and agentic workflows. This role focuses on operational reliability, workflow performance, evaluation and regression testing, configuration control, usage and cost monitoring, and the safe introduction of AI-related changes into production environments.

The ideal candidate combines cloud and application operations experience with an understanding of large language model (LLM) applications, agent behavior, and AI-specific operational risks. The engineer will collaborate with AI/ML developers, site reliability engineers, cybersecurity teams, and platform operations personnel to maintain reliable, auditable, and secure AI-enabled services.

• Monitor availability, latency, error rates, model and token usage, service costs, tool-call failures, and workflow completion metrics for production AI services.

• Integrate AI service health and agent-workflow telemetry into observability dashboards, monitoring systems, and alerting processes.

• Investigate failed, stalled, repetitive, or abnormal AI workflows and coordinate remediation with software, platform, and AI engineering teams.

• Maintain operational runbooks, escalation procedures, troubleshooting guidance, and recovery processes for AI-enabled applications.

• Distinguish infrastructure and application failures from issues associated with model behavior, prompts, retrieval sources, or agent orchestration.

• Develop and maintain repeatable evaluation scenarios for critical AI workflows, developer assistants, and model-service updates.

• Assess task completion, output quality, error patterns, inappropriate tool use, and other indicators of operational suitability.

• Coordinate regression testing before changes to models, prompts, tools, workflow logic, or retrieval sources are deployed to production.

• Analyze user feedback and operational results to identify where workflows require refinement, additional controls, or restrictions on automated execution.

• Document evaluation results, limitations, and recommendations to support evidence-based deployment decisions.

• Maintain version control and change records for prompts, agent instructions, tool definitions, model selections, evaluation datasets, and workflow configurations.

• Coordinate model and service updates with AI engineering, cybersecurity, platform operations, and architecture stakeholders.

• Support rollback to validated configurations when changes negatively affect reliability, performance, or workflow quality.

• Ensure AI-related changes follow established review, testing, approval, and deployment procedures.

• Monitor configuration drift and verify that production services remain aligned with approved configurations.

• Coordinate with cybersecurity and identity teams to verify access controls, tool permissions, logging, and auditability for production AI services.

• Support investigations involving anomalous AI behavior, unauthorized actions, or potential data-handling concerns.

• Collect and maintain operational evidence supporting security assessments, continuous monitoring, and service reviews.

• Verify that production workflows use approved models, endpoints, tools, and data sources in accordance with established policies.

• Support the implementation and validation of operational safeguards for AI-enabled functions.

• Identify repetitive operational tasks and support activities suitable for controlled AI-assisted automation.

• Measure the impact of deployed workflows on processing time, support workload, service quality, and operational efficiency.

• Collaborate with service delivery, Tier 2 support, SRE, and platform engineering teams to improve workflows based on operational demand.

• Report trends in service health, model usage, operating costs, evaluation results, and unresolved issues.

• Recommend improvements to monitoring, testing, configuration control, and operational procedures.

Location:

This position is fully onsite in Chantilly, VA.

• Active TS/SCI clearance with CI Polygraph.

• BS/BA and 5 years of experience, an additional 4 years of experience may be considered in lieu of a degree.

• Approximately 5–8 years of experience in platform operations, site reliability…