Senior AI QA Lead: ML​/GenAI Testing & Automation

Job Title: AI Quality Assurance Specialist Sr. Lead

Work Place Flexibility: Hybrid

Legal Entity: Entergy Services, LLC

• The preferred location for this role is New Orleans, LA. Other locations within Entergy’s service territory may be considered**
  *

Job Summary /Purpose

This role blends traditional software QA skills with advanced evaluation methodologies for modern AI models, including LLMs and ML-driven product features. As a member of the team, you will work closely with experienced engineers and machine learning experts to qualify and refine features powered by vision, language, and cross-modal models. You will be responsible for designing rigorous evaluation strategies for both objective and subjective ML behaviors, creating reliable automated testing pipelines, and developing LLM-driven evaluators that complement human judgement.

The ideal candidate is self-directed, creative, and comfortable with ambiguity, with strong technical and interpersonal skills. They have hands-on experience testing ML models directly, defining qualitative scoring rubrics, building reproducible evaluation frameworks, and ensuring that AI behavior is safe, consistent, and aligned with product vision.



Job Duties /Responsibilities

• Create detailed, comprehensive, and well-structured test plans and test cases for traditional and ML-based solutions. Estimate, prioritize, plan, and coordinate testing activities.
• Design, develop, and execute automation scripts. Identify, record, document, and track bugs. Perform thorough regression testing when bugs are resolved.
• Develop and maintain robust testing frameworks for machine learning models
• Develop and apply testing processes for new and existing products to meet client needs.
• Monitor debugging process results. Investigate the causes of non-conforming software. Track quality assurance metrics, such as defect densities and open defect counts.
• Collaborate with internal teams (e.g. developers, product managers, and designers) to identify testing requirements.
• Identify and prioritize quality control issues from an end-to-end perspective.
• Participate in scrum ceremonies such as refinement, demo, and release planning.

Minimum Requirements

Minimum education required of the position:
Bachelor’s degree in computer science, MIS, business administration, or other technical area of study OR equivalent work experience; advanced degree is a plus.

Minimum Experience Required Of The Position

Specialist I: 0-3 years of professional experience.

Specialist II:
At least 2 years of professional experience, at least 1 year of experience testing ML/GenAI solutions.

Specialist III:
At Least 4 years of professional experience, at least 2 years of experience testing ML/GenAI solutions.

Specialist, Sr:
At least 6 years of professional experience, at least 3 years of experience testing ML/GenAI solutions.

Sr Lead:
At least 8 years of professional experience, at least 4 years of experience testing ML/GenAI solutions.

Required

Minimum knowledge, skills and abilities required of the position:

• Strong knowledge of software QA methodologies, tools, and processes.
• Familiarity working with Agile development teams.
• Good communications skills; ability to effectively communicate verbally and in writing with both non-technical and technical staff.
• Proficient in both white box and black box testing methods.
• Ability to work both independently and as part of a team.

Desired Knowledge, Skills, And Abilities

• Working knowledge of automated testing tools.
• Highly proficient SQL and scripting skills.
• Experience testing AI/ML systems in cloud-based environments.
• Experience integrating automated AI testing into CI/CD pipelines (e.g., Git Hub Actions or similar tools).
• Experience with monitoring and logging systems for post-deployment model validation.
• Advanced experience testing generative AI systems, including LLMs for accuracy, bias, safety, and hallucinations.
• Familiarity with RAG evaluation workflows and vector databases (e.g., FAISS, Pinecone, Weaviate).
• Experience with prompt engineering, adversarial prompting, and synthetic data generation.

Any Certificates, Licenses, Etc. Required Of The Position

• Any Service Now, Scrum, or ITIL certifications are a…