AI Reliability Manager

AI Reliability Manager

Job Description:

The Risk & Fraud Product Management team is looking for an AI Reliability Manager to help ensure the agentic AI capabilities in CLEAR, our investigative platform, are accurate, trustworthy, and continuously improving. Our customers include law enforcement, financial crimes compliance teams, corporate fraud investigators, and government agencies who use our products to make consequential decisions. The quality of what our AI produces matters enormously, and this role exists to measure that quality, defend it, and drive it upward.

The AI Reliability Manager is an individual contributor who is an analytically rigorous, resourceful problem-solver who is genuinely curious about how agentic AI systems work and how to make them better. You and the reliability team will own the evaluation data and quality signals that tell us whether our AI is performing to standard, translate customer-reported issues into actionable engineering work, and serve as a trusted voice on release readiness.

This is a hands‑on role for someone who likes digging into messy output, finding the pattern in it, and turning that pattern into tangible improvements.

You will join an established evaluation program with existing gold datasets, annotation guidelines, and scoring rubrics already in production use, and you will be responsible for extending and scaling that work as our AI capabilities expand.

Key Responsibilities
Evaluation & Quality Measurement

• Own and extend the gold datasets, annotation guidelines, and scoring rubrics used to evaluate agentic AI output for accuracy, completeness, and appropriate sourcing.
• Execute evaluation cycles, scoring AI responses against established guidelines, maintaining consistency across annotators, and flagging where output falls short.
• Analyze evaluation results to identify failure patterns, quantify their impact, and recommend where engineering and data science effort should be focused.
• Test and validate AI features prior to release, and provide a clear, evidence-based point of view on release readiness.
• Train and support additional cross functional annotator resources, including globally distributed contributors who participate in evaluation cycles but are not day‑to‑day experts, ensuring they apply guidelines consistently.

Product Support & Issue Resolution

• Serve as a day‑to‑day product resource for the sales channel, answering questions about product behavior, capabilities, and known limitations, and keeping them current on the status of open issues.
• Own intake of reported issues across the CLEAR application, from AI output quality concerns to general product defects; reproduce issues, determine root cause category, and document them precisely.
• Open and manage engineering and labs tickets, and drive them to resolution alongside product, engineering, and data science partners.
• Track recurring defect themes over time and surface them to product leadership as systemic issues rather than one‑off tickets.
• Cross‑Functional Partnership
• Partner with product management, engineering, applied research, and go‑to‑market teams to translate quality findings into roadmap and prioritization decisions.
• Build an understanding of customer pain points and goals, and help identify new ways agentic AI can be applied to investigative workflows.

Required Qualifications

• Three or more years, or equivalent experience, in work requiring careful judgment about quality, such as research, analysis, editorial, audit, quality assurance, or investigative work
• Experience applying consistent standards to open‑ended work where reasonable people can disagree about what "good" looks like
• Hands‑on use of generative AI tools, with enough curiosity to have noticed how they fail, including…