AI Quality Analyst, AI & Innovation
-
IT/Tech
Job Description & How to Apply Below
Kansas City, Missouri, United States of America
At Lockton, we’re passionate about helping our people achieve their ultimate potential. Our people are curious, action-oriented and always striving to make ourselves and those around us better. We’re active listeners working to ensure understanding and problem solvers developing innovative solutions. If you can see yourself delivering excellent service to clients, giving back to our communities and being a part of our caring culture,
you belong here.
The AI Quality Analyst, AI & Innovation safeguards the quality, reliability, and measurable performance of the team’s AI data extraction processes, which convert commercial risk insurance documents — such as policies, endorsements, submissions, schedules of values, and loss runs — into structured data. Rather than building the extraction systems, this role exists to support the organization’s Enterprise AI Evaluation Strategies — the standards, methods, and processes used to determine whether AI systems are accurate, reliable, and ready for use.
The role spans both the foundation and the practice of evaluation: building and curating the ground-truth (“golden”) datasets against which AI performance is measured, and contributing directly to the evaluation processes those datasets power. Reporting to the Director, AI & Innovation, the AI Quality Analyst creates, curates, and maintains ground-truth files, helps design and run evaluations, reviews AI extraction outputs against ground truth, and delivers clear, data-driven feedback that the development team uses to advance the extraction process.
A central part of the role is engaging directly with business users: training them on the ground-truth application, guiding them through the process of producing and reviewing ground-truth data, and collecting the associated source files and validated outputs. The AI Quality Analyst maintains a comprehensive, working understanding of the extraction processes to define what “correct” looks like and to safeguard extraction quality end to end.
The role is well suited to someone who brings a working knowledge of commercial risk insurance, is familiar with AI/ML and large language model concepts, and is skilled at quality review, data stewardship, and stakeholder engagement — without requiring hands-on AI/ML development experience.
- Own, create, and maintain the curated ground-truth (“golden”) datasets and ground-truth labels that developers rely on to measure, benchmark, and improve AI extraction quality — the foundation of the team’s evaluation work.
- Contribute to the design, execution, and continuous improvement of the organization’s Enterprise AI Evaluation Strategies, making sure ground-truth data is built to serve the evaluations it supports rather than simply collected and stored.
- Help design evaluation plans and test sets for AI extraction use cases, selecting representative and edge‑case samples across commercial risk insurance document types, lines of business, and levels of complexity.
- Participate in evaluation cycles — baseline, regression, and pre‑release testing — whenever models, prompts, or processes change, and help define the acceptance thresholds that determine whether changes are ready for production.
- Help calibrate automated evaluation methods, such as LLM-as-judge scoring, against human‑reviewed ground truth to confirm that those methods can be trusted at scale.
- Contribute to enterprise evaluation standards, templates, and playbooks so evaluation practices are consistent and reusable across AI initiatives beyond data extraction.
- Define, document, and uphold clear standards and annotation guidelines for what constitutes a correct extraction, so quality can be measured…
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction