AI Automation Engineer

Job Title:

AI Test Engineer

Location:

Vienna, VA
Type:
Contract
Work Model:
Hybrid – onsite and remote

Hours:

40.0

• Help design, develop, and maintain AI testing and evaluation frameworks for generative AI, predictive AI, and AI-powered developer experience (Dev Ex) solutions.
• Create and execute automated test suites that validate AI model behavior, output quality, accuracy, reliability, and performance.
• Develop automated evaluation pipelines that integrate with CI/CD workflows and support continuous AI quality validation.
• Perform functional, regression, performance, and scenario‑based testing of AI‑enabled applications and services.
• Analyze AI-generated outputs to identify defects, inconsistencies, hallucinations, bias, performance issues, and other quality concerns.
• Build and maintain test datasets, benchmark scenarios, prompt libraries, and evaluation artifacts used to assess AI systems.
• Collaborate with software engineers, AI engineers, and platform teams to improve model quality and testing coverage.
• Define, measure, and report AI quality metrics such as accuracy, relevance, latency, consistency, robustness, and user satisfaction.
• Support root‑cause analysis, troubleshooting, and validation activities for AI‑related defects and incidents.
• Document test strategies, test results, evaluation methodologies, and recommendations for technical and non‑technical stakeholders.
• Contribute to AI governance and responsible AI efforts by supporting evaluation activities related to reliability, fairness, transparency, and safety.
• Support delivery of strategic Dev Ex initiatives and AI-enabled platform capabilities across the software development lifecycle.

• Bachelor's degree in Computer Science, Information Systems, Software Engineering, Data Science, or a related technical field.
• 3+ years of experience in software testing, quality engineering, test automation, or AI/ML testing.
• Experience designing and executing automated test strategies for complex software applications.
• Strong programming and scripting skills in Python.
• Experience building automated test frameworks and test pipelines.
• Experience working with APIs, automated integration testing, and CI/CD environments.
• Understanding of machine learning, generative AI, large language models (LLMs), or AI-powered applications.
• Experience performing data validation, test result analysis, and defect investigation.
• Familiarity with modern testing methodologies, quality engineering practices, and software delivery life cycles.
• Knowledge of statistical analysis and evaluation techniques used to validate system performance.
• Experience using source control systems and collaborative development practices.
• Strong analytical, problem‑solving, and troubleshooting skills.
• Excellent written and verbal communication skills.
• Ability to work independently in a fast‑paced, agile environment while collaborating effectively across teams.

• Experience testing generative AI applications, LLM‑based solutions, or Retrieval‑Augmented Generation (RAG) systems.
• Experience with AI evaluation frameworks such as Deep Eval, Ragas, Lang Smith, or similar tools.
• Knowledge of LLM observability, monitoring, and traceability practices.
• Experience working with Azure AI, Git Hub Copilot, OpenAI, Hugging Face, Lang Chain, or comparable AI platforms.
• Familiarity with prompt engineering, benchmark dataset creation, and AI output validation techniques.
• Understanding of Responsible AI principles and AI governance frameworks.
• Experience with performance testing, scalability testing, and reliability engineering.
• Experience supporting developer platforms, Dev Ex initiatives, or software engineering productivity tools.
• Experience working within enterprise or regulated technology…