Customer AI Operations Engineer

Overview

Location: Hybrid Preferred (Minneapolis, MN or Denver, CO) | Remote Considered
Employment Type: Full-time

Are you passionate about ensuring AI solutions deliver real business value after deployment? Trissential is seeking a Customer AI Operations Engineer to join our client's team and take ownership of the operational success of enterprise AI agents, virtual assistants, and generative AI solutions in production. This is a unique opportunity to influence AI quality, governance, adoption, and business outcomes while partnering with product, engineering, operations, and business stakeholders.

What's in It for You?

• Be the operational owner of cutting-edge AI solutions used across a large enterprise environment
• Drive measurable improvements in AI quality, adoption, customer experience, and ROI
• Collaborate with Product, AI Engineering, Operations, Security, Governance, and business SMEs
• Gain exposure to Responsible AI practices, enterprise governance programs, and AI observability platforms
• Work in a highly visible role where your insights directly influence AI strategy and operational outcomes
• Opportunity to help define long-term AI operational standards and potentially transition into a permanent role

Your Role & Responsibilities

• Own the operational performance of AI-powered solutions after implementation and hypercare
• Monitor AI quality, response accuracy, relevance, compliance, adoption, and business outcomes
• Develop and maintain operational scorecards covering containment, satisfaction, escalation rates, retrieval effectiveness, response quality, latency, and utilization
• Review transcripts and AI conversations to evaluate accuracy, grounding, compliance, and overall response quality
• Investigate hallucinations, retrieval failures, misrouted interactions, prompt issues, knowledge gaps, and workflow breakdowns
• Manage and optimize Knowledge Hub performance through content gap analysis, search quality evaluations, source validation, and freshness monitoring
• Support Responsible AI monitoring activities including privacy validation, governance reporting, audit readiness, bias reviews, and guardrail compliance
• Coordinate incident investigations and identify root causes across business and technical teams
• Translate operational findings into prioritized backlog items for Product Owners and AI Engineering teams
• Measure and report value realization, including productivity gains, self-service adoption, customer outcomes, utilization metrics, and ROI
• Develop operational procedures, governance standards, and knowledge-transfer documentation to support long-term sustainability

Skills & Experience You Should Possess

• 5+ years of experience in technology operations, analytics, product operations, quality engineering, knowledge management, conversational AI, or a related field
• At least 2 years supporting AI agents, conversational AI platforms, virtual assistants, or generative AI solutions in a production environment
• Experience evaluating AI responses through transcript reviews, response grading, hallucination analysis, and quality scorecards
• Strong operational analytics experience with dashboards, KPIs, performance reporting, and trend analysis
• Knowledge of retrieval operations, search effectiveness, knowledge management, source validation, and content-gap analysis
• Experience investigating production incidents and driving root-cause analysis efforts
• Understanding of Responsible AI principles, governance frameworks, privacy considerations, and compliance monitoring
• Excellent communication skills with the ability to explain AI quality issues and business impacts to diverse stakeholders
• Strong documentation, organizational, and continuous-improvement capabilities

Bonus Points If You Have

• Experience supporting AI agents within…