AI Automation Test Engineer

Job Title:
Automation Test Engineer with AI

Location: Chennai (Work from Home) Overlapping / Night shift

Experience: 3+

LLM Testing, Chatbot Testing, Voice Bot Testing, Conversational AI Testing, Prompt Validation, RAG Testing, Hallucination Testing

Responsibilities

• Agentic Benchmark Design:
  Generate, import, create, extend, and manage benchmarks for non-deterministic LLM agents, validating behaviour and responses.
• Functional & Regression Testing:
  Grow benchmarks and create new ones as new functionality or issues are uncovered.
• Benchmarking Automation Support:
  Contribute to streamlining the benchmarking process in alignment with product development work.

Collaboration:

Work closely with developers, product managers, and senior QA engineers to ensure test coverage aligns with product requirements.

• Defect Identification & Reporting:
  Log detailed defect reports and work with engineering teams to ensure timely resolution.
• Process & Traceability:
  Maintain requirement-to-test traceability and document detailed test plans and coverage to support audits/reviews for OEM programs.
• Release Reporting:
  Produce per-release quality reports as benchmarks evolve and grow over time.
• Performance & Load Testing Support:
  Assist in load and performance testing using tools such as JMeter.
• Agile Testing & Continuous Improvement:
  Participate in Agile development cycles, contributing to continuous process improvements.

Qualifications Basic Qualifications (Non-Negotiable)

• 3+ years of experience in software/product quality assurance.
• Confident usage of Bash and Python.
• Experience with manual testing and test case execution.
• Basic familiarity with automated testing frameworks such as Playwright or Selenium.
• Experience with testing web protocols (Web Socket message payload) GEN AI / Agentic AI Testing
• Basic knowledge of load and performance testing (e.g., JMeter).
• Strong analytical and problem-solving abilities.
• Systems thinking.
• Familiarity with prompt evaluation, function-calling, and tool-use testing.

Preferred Skills

• Exposure to test automation scripting using Python.
• Experience with CI/CD integration.
• Familiarity with Conversational AI platforms and NLP-based applications.
• Experience with test management and result tracking.
• Experience testing conversational-agent systems and evaluating non deterministic outputs.