Quality Engineering Lead; AI​/ML​/LLM

Position: Quality Engineering Lead (AI/ML/LLM)
Job Title: Quality Engineering Lead (AI/ML/LLM)
Location: Chicago, IL / /Minneapolis, MN (onsite)
Job Type: Full-Time
Salary Range: $130,000 $ 140,000 per year
Role Summary
We are seeking a highly skilled Quality Engineering Lead to establish and lead the AI Quality Engineering function within a growing AI delivery organization. The ideal candidate will possess strong expertise in AI/ML/LLM testing, validation frameworks, model evaluation methodologies, quality governance, and enterprise-scale AI deployment practices. This role will be responsible for defining quality standards, developing evaluation frameworks, implementing validation processes, and ensuring AI systems meet reliability, safety, explainability, and governance requirements.
Must Have Technical/Functional Skills
8+ years of experience in Quality Engineering, Software Testing, or Quality Assurance leadership.
Hands-on experience with AI/ML/LLM Quality Engineering, testing, validation, and evaluation frameworks.
Strong understanding of deterministic and non-deterministic AI system evaluation methodologies.
Experience designing and implementing validation frameworks, golden datasets, and test harnesses.
Expertise in scalable testing strategies for enterprise AI and Generative AI platforms.
Experience embedding quality engineering practices throughout the AI software development lifecycle (AI SDLC).
Strong knowledge of model evaluation, monitoring, drift detection, and continuous quality improvement.
Experience evaluating AI agents, autonomous systems, and LLM-based applications.
Understanding of agent maturity models and autonomy thresholds.
Knowledge of responsible AI, governance, explainability, fairness, and compliance frameworks.
Strong analytical, problem-solving, and leadership skills.
Excellent communication and stakeholder management capabilities.
Roles & Responsibilities
Build and establish the AI Quality Engineering function as a strategic capability within the enterprise AI organization.
Define and implement quality engineering standards, methodologies, and governance processes for AI systems.
Develop evaluation frameworks for both deterministic and non-deterministic AI model outputs.
Establish validation criteria, golden datasets, benchmark datasets, and automated testing frameworks.
Design and implement test harnesses to support model validation, prompt testing, regression testing, and release readiness.
Drive adoption of AI-powered Quality Engineering capabilities to improve testing efficiency and quality outcomes.
Collaborate with engineering, architecture, data science, and synthetic data teams to embed quality controls throughout the AI lifecycle.
Define and monitor key quality metrics, KPIs, and maturity assessments for AI systems and QE processes.
Implement continuous monitoring and quality assurance processes for deployed AI models and agents.
Evaluate model performance, reliability, safety, explainability, and operational readiness.
Establish governance frameworks for AI model releases, prompt updates, and autonomous agent deployments.

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.