AI Test & Quality Engineer; Hybrid

Job Title:

AI Test Engineer

Location:

Vienna, VA
Type:
Contract
Work Model:
Hybrid – onsite and remote

Hours:

40.0

Responsibilities

• Help design, develop, and maintain AI testing and evaluation frameworks for generative AI, predictive AI, and AI-powered developer experience (Dev Ex) solutions.

• Create and execute automated test suites that validate AI model behavior, output quality, accuracy, reliability, and performance.

• Develop automated evaluation pipelines that integrate with CI/CD workflows and support continuous AI quality validation.

• Perform functional, regression, performance, and scenario-based testing of AI-enabled applications and services.

• Analyze AI-generated outputs to identify defects, inconsistencies, hallucinations, bias, performance issues, and other quality concerns.

• Build and maintain test datasets, benchmark scenarios, prompt libraries, and evaluation artifacts used to assess AI systems.

• Collaborate with software engineers, AI engineers, and platform teams to improve model quality and testing coverage.

• Define, measure, and report AI quality metrics such as accuracy, relevance, latency, consistency, robustness, and user satisfaction.

• Support root-cause analysis, troubleshooting, and validation activities for AI-related defects and incidents.

• Document test strategies, test results, evaluation methodologies, and recommendations for technical and non-technical stakeholders.

• Contribute to AI governance and responsible AI efforts by supporting evaluation activities related to reliability, fairness, transparency, and safety.

• Support delivery of strategic Dev Ex initiatives and AI-enabled platform capabilities across the software development lifecycle.

Requirements

• Bachelor's degree in Computer Science, Information Systems, Software Engineering, Data Science, or a related technical field.

• 3+ years of experience in software testing, quality engineering, test automation, or AI/ML testing.

• Experience designing and executing automated test strategies for complex software applications.

• Strong programming and scripting skills in Python.

• Experience building automated test frameworks and test pipelines.

• Experience working with APIs, automated integration testing, and CI/CD environments.

• Understanding of machine learning, generative AI, large language models (LLMs), or AI-powered applications.

• Experience performing data validation, test result analysis, and defect investigation.

• Familiarity with modern testing methodologies, quality engineering practices, and software delivery life cycles.

• Knowledge of statistical analysis and evaluation techniques used to validate system performance.

• Experience using source control systems and collaborative development practices.

• Strong analytical, problem-solving, and troubleshooting skills.

• Excellent written and verbal communication skills.

• Ability to work independently in a fast-paced, agile environment while collaborating effectively across teams.

Preferred Qualifications:

• Experience testing generative AI applications, LLM-based solutions, or Retrieval-Augmented Generation (RAG) systems.

• Experience with AI evaluation frameworks such as Deep Eval, Ragas, Lang Smith, or similar tools.

• Knowledge of LLM observability, monitoring, and traceability practices.

• Experience working with Azure AI, Git Hub Copilot, OpenAI, Hugging Face, Lang Chain, or comparable AI platforms.

• Familiarity with prompt engineering, benchmark dataset creation, and AI output validation techniques.

• Understanding of Responsible AI principles and AI governance frameworks.

• Experience with performance testing, scalability testing, and reliability engineering.

• Experience supporting developer platforms, Dev Ex initiatives, or software engineering…