AI QA Engineer Job New York City area,New York USA,Software Development

AI QA Engineer

We are seeking an AI QA Engineer to ensure the quality, reliability, safety, and performance of AI-powered applications and machine learning solutions. The ideal candidate will design and execute testing strategies for AI systems, including large language model (LLM) applications, generative AI features, machine learning models, and AI-driven workflows. This role requires expertise in traditional software quality assurance combined with an understanding of AI model behavior, prompt engineering, and automated testing.

Key Responsibilities

Design and implement test strategies for AI and machine learning applications.
Develop and maintain automated test frameworks for AI-powered systems.
Validate LLM responses for accuracy, consistency, relevance, and safety.
Create and manage test datasets, prompts, and evaluation benchmarks.
Test AI features for functionality, usability, performance, security, and reliability.
Identify, document, and track defects using bug management tools.
Perform regression, integration, API, end-to-end, and performance testing.
Evaluate AI systems for hallucinations, bias, toxicity, and prompt injection vulnerabilities.
Collaborate with AI engineers, data scientists, product managers, and developers to resolve issues.
Analyze model outputs and recommend improvements to prompts or workflows.
Monitor AI application performance after deployment.
Ensure compliance with organizational quality standards and AI governance policies.

Required Qualifications

Bachelor's degree in Computer Science, Information Technology, Software Engineering, or a related field.
3–5+ years of experience in software quality assurance or test automation.
Experience testing web, mobile, or API-based applications.
Familiarity with AI, machine learning, or generative AI applications.
Strong understanding of software testing methodologies and QA best practices.
Experience with Agile or Scrum development environments.

Preferred Qualifications

Experience testing LLM-based or generative AI applications.
Knowledge of AI evaluation frameworks and benchmarking techniques.
Familiarity with prompt engineering and AI model behavior.
AI or cloud certifications (AWS, Microsoft Azure, Google Cloud, etc.).
Experience with MLOps or AI deployment pipelines.

Technical Skills

Manual and automated testing
Test case design and execution
API testing (Postman, REST Assured)
Automation tools (Selenium, Playwright, Cypress)
Programming (Python, Java, JavaScript, or C#)
SQL and database validation
Git and CI/CD pipelines
Test management tools (Jira, Test Rail, Zephyr)
Performance testing (JMeter, k6)
AI model evaluation techniques
Prompt engineering
LLM testing and validation
AI safety testing (hallucinations, bias, toxicity, prompt injection)
Basic machine learning concepts
JSON and REST APIs

Soft Skills

Analytical thinking
Strong attention to detail
Problem-solving
Communication and documentation
Collaboration with cross-functional teams
Critical thinking
Continuous learning
Time management

Preferred Experience

AI-powered enterprise applications
Generative AI products
Machine learning platforms
SaaS applications
Conversational AI or chatbot testing
Cloud-native applications

Success Metrics

Defect detection rate
Test automation coverage
AI response quality and accuracy
Reduction in production defects
Test execution efficiency
AI safety and compliance metrics
Release quality
Customer satisfaction
Mean time to detect (MTTD) and resolve (MTTR) issues

Nice-to-Have Skills

Lang Chain or similar AI orchestration frameworks
AI evaluation frameworks (e.g., Deep Eval, Ragas, Lang Smith)
Vector databases
Docker and Kubernetes
MLOps tools
Cloud platforms (AWS, Azure, Google Cloud)
Data annotation and synthetic data generation
Security testing for AI systems
Accessibility testing
Experience with Retrieval-Augmented Generation (RAG) systems