AI​/ML Operations & Assurance Engineer

Responsibilities We are seeking an AI/ML Operations & Assurance Engineer to monitor, evaluate, and maintain production AI-enabled applications, foundation-model integrations, and agentic workflows. This role focuses on operational reliability, workflow performance, evaluation and regression testing, configuration control, usage and cost monitoring, and the safe introduction of AI-related changes into production environments.

The ideal candidate combines cloud and application operations experience with an understanding of large language model (LLM) applications, agent behavior, and AI-specific operational risks. The engineer will collaborate with AI/ML developers, site reliability engineers, cybersecurity teams, and platform operations personnel to maintain reliable, auditable, and secure AI-enabled services.

Responsibilities Production Monitoring and Operations Monitor availability, latency, error rates, model and token usage, service costs, tool-call failures, and workflow completion metrics for production AI services.

Integrate AI service health and agent-workflow telemetry into observability dashboards, monitoring systems, and alerting processes.

Investigate failed, stalled, repetitive, or abnormal AI workflows and coordinate remediation with software, platform, and AI engineering teams.

Maintain operational runbooks, escalation procedures, troubleshooting guidance, and recovery processes for AI-enabled applications.

Distinguish infrastructure and application failures from issues associated with model behavior, prompts, retrieval sources, or agent orchestration.

AI Evaluation and Operational Assurance Develop and maintain repeatable evaluation scenarios for critical AI workflows, developer assistants, and model-service updates.

Assess task completion, output quality, error patterns, inappropriate tool use, and other indicators of operational suitability.

Coordinate regression testing before changes to models, prompts, tools, workflow logic, or retrieval sources are deployed to production.

Analyze user feedback and operational results to identify where workflows require refinement, additional controls, or restrictions on automated execution.

Document evaluation results, limitations, and recommendations to support evidence-based deployment decisions.

Change and Configuration Management Maintain version control and change records for prompts, agent instructions, tool definitions, model selections, evaluation datasets, and workflow configurations.

Coordinate model and service updates with AI engineering, cybersecurity, platform operations, and architecture stakeholders.

Support rollback to validated configurations when changes negatively affect reliability, performance, or workflow quality.

Ensure AI-related changes follow established review, testing, approval, and deployment procedures.

Monitor configuration drift and verify that production services remain aligned with approved configurations.

Security, Audit, and Compliance Support Coordinate with cybersecurity and identity teams to verify access controls, tool permissions, logging, and auditability for production AI services.

Support investigations involving anomalous AI behavior, unauthorized actions, or potential data-handling concerns.

Collect and maintain operational evidence supporting security assessments, continuous monitoring, and service reviews.

Verify that production workflows use approved models, endpoints, tools, and data sources in accordance with established policies.

Support the implementation and validation of operational safeguards for AI-enabled functions.

Operational Improvement and Reporting Identify repetitive operational tasks and support activities suitable for controlled AI-assisted automation.

Measure the impact of deployed workflows on processing time, support workload, service quality, and operational efficiency.

Collaborate with service delivery, Tier 2 support, SRE, and platform engineering teams to improve workflows based on operational demand.

Report trends in service health, model usage, operating costs, evaluation results, and unresolved issues.

Recommend improvements to monitoring, testing, configuration control, and operational procedures.

Location:

This position is fully onsite in Chantilly, VA.Qualifications

Required Qualifications Active TS/SCI clearance with CI Polygraph.

BS/BA and 5 years of experience, an additional 4 years of experience may be considered in lieu of a degree.

Approximately 5–8 years of experience in platform operations, site…