AI Automation Engineer - Virtual

Job Title & Location

AI Automation Engineer - Virtual
Alight Solutions – Lincolnshire, IL
Full-time
Medical, Life, Retirement, PTO
Posted 5 hours ago

As the AI Automation Engineer in the AICoE, you will be instrumental in standing up the initial automation framework for AI Quality. You’ll work closely with the AI Quality Lead and developers to build, run, and scale automated test suites for AI models and agents.

• Develop and maintain automated test frameworks for LLM, RAG, agentic, and classic ML systems
• Implement golden/adversarial test cases (functional, safety, bias, prompt injection, refusal correctness, tool use)
• Integrate evaluation tools (promptfoo, Ragas) into CI/CD pipelines and support batch quality assessments
• Author and validate OpenAPI/JSON Schema contracts for model inputs/outputs; enforce schema checks in pipelines
• Instrument observability metrics (latency, success rate, cost, drift) using Open Telemetry and export to dashboards
• Support data QA, lineage tracking, and drift monitoring (Evidently, Open Lineage)
• Collaborate with AI developers, SDETs, and SRE to troubleshoot, optimize, and document automation processes
• Contribute to red‑team playbooks and security test books mapped to OWASP/ATLAS risks

• Experience in test automation, software engineering, or quality assurance (AI/ML experience is a plus, but not required)
• Proficiency in Python, JavaScript/Node.js, or similar programming languages
• Familiarity with CI/CD, version control, and cloud-based deployment environments
• Experience with test automation tools (promptfoo, Ragas, Postman)
• Understanding of AI model evaluation metrics (faithfulness, answer relevancy, hallucination, refusal correctness)
• Basic knowledge of fairness, safety, and security testing for AI systems
• Experience with AI infrastructure, orchestration, data integrity, deployments, and model registry practices
• Interest in Responsible AI, bias mitigation, and regulatory compliance
• Bachelor’s degree in Computer Science, Engineering, or related field, or equivalent practical experience

We offer programs and plans for a healthy mind, body, wallet, and life because it’s important our benefits care for the whole person. Options include a variety of health coverage options, wellbeing and support programs, retirement, vacation and sick leave, maternity, paternity & adoption leave, continuing education and training as well as several voluntary benefit options.

Alight is an Equal Employment Opportunity employer and does not discriminate against anyone based on sex, race, color, religion, creed, national origin, ancestry, age, physical or mental disability, medical condition, pregnancy, marital or domestic partner status, citizenship, military or veteran status, sexual orientation, gender, gender identity or expression, genetic information, or any other legally protected characteristics or conduct covered by federal, state, or local law.

In addition, we take affirmative action to employ disabled persons, disabled veterans, and other covered veterans. Alight provides reasonable accommodations to the known limitations of otherwise qualified employees and applicants for employment with disabilities and sincerely held religious beliefs, practices and observances unless doing so would result in undue hardship. Applicants for employment may request a reasonable accommodation/modification by contacting their recruiter.

By applying for a position with Alight, you understand that, should you be made an offer, it will be contingent on your undergoing and successfully completing a background check consistent with Alight's employment policies. Background checks may include SSN/SIN validation, education verification, employment verification, and criminal check, search against global sanctions and government watch lists, credit check, and/or drug test. You will be notified during the hiring process which checks are required by the position.