AI Quality Lead - Virtual

About Alight

At Alight, we believe a company's success starts with its people. We champion people, grow with purpose, and encourage colleagues to “Be Alight.” Our values focus on empathy, inspiration, integrity, and empowerment. We help clients build a healthy, financially secure workforce with a unified benefits ecosystem across health, wealth, wellbeing, absence management, and navigation.

As the AI Quality Lead for the Artificial Intelligence Center of Excellence (AI CoE), you will design and drive the end‑to‑end AI Quality program—from strategy to operational execution—combining leadership with deep technical expertise.

• Design and implement the unified AI Quality strategy and SOP, scaling controls by risk tier.
• Define and own success metrics, risk tiering, acceptance criteria, and operating thresholds for AI solutions.
• Lead the development of golden/adversarial test suites (prompt injection, misuse, safety, bias, functional automation).
• Oversee the creation and maintenance of schema‑level UI/API tests, agent contracts, and observability instrumentation.
• Own the org‑wide evaluation harnesses (e.g., promptfoo, Tru Lens, Ragas), CI gating, and red‑team playbooks.
• Manage independent spot checks, factsheets, registry entries, and evidence for governance and audit.
• Collaborate with Product Owners, Developers, SRE/Observability, and Governance to ensure compliance and operational excellence.
• Drive continuous improvement in fairness, explainability, safety, security, and operational SLOs.
• Mentor and grow the AI Quality team, including onboarding and coaching junior engineers.
• Experience with LLM, RAG, agentic, or classic ML systems.
• Familiarity with test automation frameworks, CI/CD pipelines, and evaluation tools (promptfoo, Tru Lens, Ragas).
• Knowledge of fairness, explainability, safety, and security standards (e.g., OWASP Top 10, MITRE ATLAS, Fairlearn, AIF
  360).
• Experience with data integrity, lineage, and observability (Open Telemetry, New Relic, Open Lineage).
• Familiarity with AI infrastructure, orchestration, deployments, and model registry practices.
• Knowledge of international standards and regulatory frameworks for Responsible AI.

• Experience leading or building quality engineering or testing programs (AI/ML experience is a plus).
• Ability to design and implement test strategies, automation frameworks, and quality standards.
• Excellent communication and leadership skills; ability to work cross‑functionally and influence stakeholders.
• Experience with LLM, RAG, agentic, or classic ML systems.
• Familiarity with test automation frameworks, CI/CD pipelines, and evaluation tools (promptfoo, Tru Lens, Ragas).
• Knowledge of fairness, explainability, safety, and security standards (e.g., OWASP Top 10, MITRE ATLAS, Fairlearn, AIF
  360).
• Experience with data integrity, lineage, and observability (Open Telemetry, New Relic, Open Lineage).
• Familiarity with AI infrastructure, orchestration, deployments, and model registry practices.
• Knowledge of international standards and regulatory frameworks for Responsible AI.
• Bachelor’s or Master’s degree in Computer Science, Engineering, Data Science, or related field, or equivalent experience.

• Comprehensive total rewards package, including health coverage, wellbeing and support programs, retirement, vacation and sick leave.
• Maternity, paternity & adoption leave.
• Continuing education and training.
• Flexible work arrangements and recognition as a Top 100 Company for Remote Jobs.
• Volunteer benefit options.

Salary range: $100,000 – $160,000 per year.

Alight is an equal employment opportunity employer and does not discriminate on the basis of sex, race, color, religion, creed, national origin, ancestry, age, physical or mental disability, medical condition, pregnancy, marital or domestic partner status, citizenship, military or veteran status, sexual orientation, gender, gender identity, or expression, genetic information, or any other protected characteristic. We take affirmative action to employ disabled and veteran employees.

All virtual interviews will be conducted on video.

Applicants must have work authorization that does not require future sponsorship of a visa for employment in the employing country.