AI Test Engineers

Job#: 3054790

Job Description:

AI Test Engineers

Location: Pensacola, FL, Vienna, VA, Winchester, VA, San Deigo, CA (Hybrid)

Alternate

Location:

Open to Remote (U.S.)

We are seeking a motivated AI Test Engineer to help establish and scale our AI testing and evaluation capability. This role will work closely with AI engineers, platform teams, and product stakeholders to build automated testing frameworks, evaluate AI-driven solutions, and ensure the quality, reliability, and performance of AI-enabled products. The ideal candidate is a hands-on quality engineer who enjoys building testing capabilities rather than simply executing predefined test cases.

They are curious about how AI systems behave and are passionate about improving reliability through automation.

• Design, develop, and maintain AI testing and evaluation frameworks for generative AI, predictive AI, and AI-powered developer experience (Dev Ex) solutions.
• Create and execute automated test suites that validate AI model behavior, output quality, accuracy, reliability, and performance.
• Develop automated evaluation pipelines that integrate with CI/CD workflows and support continuous AI quality validation.
• Perform functional, regression, performance, and scenario-based testing of AI-enabled applications and services.
• Analyze AI-generated outputs to identify defects, inconsistencies, hallucinations, bias, and performance issues.
• Build and maintain test datasets, benchmark scenarios, prompt libraries, and evaluation artifacts used to assess AI systems.
• Collaborate with software engineers, AI engineers, and platform teams to improve model quality and testing coverage.
• Define, measure, and report AI quality metrics such as accuracy, relevance, latency, consistency, robustness, and user satisfaction.
• Support root-cause analysis, troubleshooting, and validation activities for AI-related defects and incidents.
• Document test strategies, test results, evaluation methodologies, and recommendations for stakeholders.
• Contribute to AI governance and responsible AI efforts by supporting evaluation activities related to reliability, fairness, transparency, and safety.
• Support delivery of strategic Dev Ex initiatives and AI-enabled platform capabilities across the software development lifecycle.

Education: Bachelor's degree in Computer Science, Information Systems, Software Engineering, Data Science, or a related technical field.

Experience: 3+ years of experience in software testing, quality engineering, test automation, or AI/ML testing. Experience designing and executing automated test strategies for complex software applications. Experience performing data validation, test result analysis, and defect investigation. Experience using source control systems and collaborative development practices.

Technical

Skills:

Strong programming and scripting skills in Python. Experience building automated test frameworks and test pipelines. Experience working with APIs, automated integration testing, and CI/CD environments. Understanding of machine learning, generative AI, and large language models (LLMs). Familiarity with modern testing methodologies, quality engineering practices, and software delivery life cycles. Knowledge of statistical analysis and evaluation techniques used to validate system performance.

• Experience testing generative AI applications, LLM-based solutions, or Retrieval-Augmented Generation (RAG) systems.
• Experience with AI evaluation frameworks such as Deep Eval, Ragas, Lang Smith, or similar tools.
• Knowledge of LLM observability, monitoring, and traceability practices.
• Experience working with Azure AI, Git Hub Copilot, OpenAI, Hugging Face, Lang Chain, or comparable AI platforms.
• Familiarity with prompt engineering, benchmark dataset creation, and AI output validation techniques.
• Understanding of Responsible AI principles and AI governance frameworks.
• Experience with performance testing, scalability testing, and reliability engineering.
• Experience supporting developer platforms, Dev Ex initiatives, or software engineering productivity tools.
• Experience working within enterprise or regulated technology environments.

Everforth Apex is a world-class IT services company that serves thousands of clients across the globe. When you join Everforth Apex, you become part of a team that values innovation, collaboration, and continuous learning. We offer quality career resources, training, certifications, development…