AI Quality Assurance Specialist Sr. Lead

AI Quality Assurance Specialist Sr. Lead

Job Title:

AI Quality Assurance Specialist Sr. Lead Work Place Flexibility:
Hybrid Legal Entity:
Entergy Services, LLC
*** The preferred location for this role is New Orleans, LA. Other locations within Entergy's service territory may be considered***

JOB SUMMARY/

PURPOSE:

This role blends traditional software QA skills with advanced evaluation methodologies for modern AI models, including LLMs and ML-driven product features. As a member of the team, you will work closely with experienced engineers and machine learning experts to qualify and refine features powered by vision, language, and cross-modal models. You will be responsible for designing rigorous evaluation strategies for both objective and subjective ML behaviors, creating reliable automated testing pipelines, and developing LLM-driven evaluators that complement human judgement.

The ideal candidate is self-directed, creative, and comfortable with ambiguity, with strong technical and interpersonal skills. They have hands-on experience testing ML models directly, defining qualitative scoring rubrics, building reproducible evaluation frameworks, and ensuring that AI behavior is safe, consistent, and aligned with product vision.

JOB DUTIES/RESPONSIBILITIES:

• Create detailed, comprehensive, and well-structured test plans and test cases for traditional and ML-based solutions. Estimate, prioritize, plan, and coordinate testing activities.
• Design, develop, and execute automation scripts. Identify, record, document, and track bugs. Perform thorough regression testing when bugs are resolved.
• Develop and maintain robust testing frameworks for machine learning models
• Develop and apply testing processes for new and existing products to meet client needs.
• Monitor debugging process results. Investigate the causes of non-conforming software. Track quality assurance metrics, such as defect densities and open defect counts.
• Collaborate with internal teams (e.g. developers, product managers, and designers) to identify testing requirements.
• Identify and prioritize quality control issues from an end-to-end perspective.
• Participate in scrum ceremonies such as refinement, demo, and release planning.

MINIMUM REQUIREMENTS:

• Minimum education required of the position:
  Bachelor's degree in computer science, MIS, business administration, or other technical area of study OR equivalent work experience; advanced degree is a plus.
• Minimum experience required of the position:
  Specialist I: 0-3 years of professional experience. Specialist II:
  At least 2 years of professional experience, at least 1 year of experience testing ML/GenAI solutions. Specialist III:
  At Least 4 years of professional experience, at least 2 years of experience testing ML/GenAI solutions. Specialist, Sr:
  At least 6 years of professional experience, at least 3 years of experience testing ML/GenAI solutions. Consultant:
  At least 8 years of professional experience, at least 4 years of experience testing ML/GenAI solutions. Consultant, Sr:
  At least 10 years of professional experience, at least 5 years of experience testing ML/GenAI solutions.
• Minimum knowledge, skills and abilities required of the position:
• Strong knowledge of software QA methodologies, tools, and processes.
• Familiarity working with Agile development teams.
• Good communications skills; ability to effectively communicate verbally and in writing with both non-technical and technical staff.
• Proficient in both white box and black box testing methods.
• Ability to work both independently and as part of a team.
• Desired knowledge, skills, and abilities:
• Working knowledge of automated testing tools.
• Highly proficient SQL and scripting skills.
• Experience testing AI/ML systems in cloud-based environments.
• Experience integrating automated AI testing into CI/CD pipelines (e.g., Git Hub Actions or similar tools).
• Experience with monitoring and logging systems for post-deployment model validation.
• Advanced experience testing generative AI systems, including LLMs for accuracy, bias, safety, and hallucinations.
• Familiarity with RAG evaluation workflows and vector databases (e.g., FAISS, Pinecone, Weaviate).
• Experience with prompt…