Quality Engineer/SDET – AI Chatbot Validation
Listed on 2026-10-21
-
Quality Assurance - QA/QC
AI QA / Validation Engineer, IT QA Tester / Automation
Primary
Skills:
AI/LLM Chatbot Testing, Test Automation (Python/Selenium/Playwright), API Testing & Quality Assurance
Location: Toronto, ON
Work Model: Onsite
Job Type: Full Time
We are seeking an experienced Quality Engineer (QE) / Software Development Engineer in Test (SDET) to validate AI-powered chatbots and conversational applications. The role focuses on ensuring chatbot responses are accurate, relevant, complete, consistent, and aligned with expected behavior.
The ideal candidate will have strong QA and test automation experience, familiarity with AI/LLM-based applications, and the ability to identify hallucinations, incorrect responses, formatting issues, and knowledge gaps. The candidate will collaborate with developers, AI engineers, and content teams to improve chatbot quality and reliability throughout the development lifecycle.
Key Responsibilities- Validate chatbot responses for accuracy, relevance, completeness, consistency, and contextual correctness.
- Design, develop, and execute test scenarios, test cases, and regression test suites for AI-powered chatbots.
- Identify and document hallucinations, incorrect answers, formatting defects, and knowledge gaps.
- Perform chatbot conversation analysis and recommend improvements to response quality and user experience.
- Develop and maintain automated test scripts and frameworks for chatbot validation where feasible.
- Conduct API testing and validate integrations with supporting services and knowledge sources.
- Collaborate with developers, AI engineers, and content teams to investigate and resolve defects.
- Track, document, and report defects, test results, quality metrics, and testing progress.
- Support User Acceptance Testing (UAT), release validation, and production quality monitoring.
- Apply prompt engineering concepts to design effective test scenarios and evaluate chatbot behavior.
- Participate in Agile ceremonies and contribute to continuous quality improvement.
- 5+ years of experience in Quality Assurance, SDET, software testing, or test automation.
- Experience testing AI, LLM, Generative AI, chatbots, virtual assistants, or conversational platforms.
- Understanding of prompt engineering concepts and AI response evaluation.
- Experience with API testing and test automation frameworks.
- Hands-on experience with Python, Selenium, Playwright, PyTest, or similar testing tools.
- Experience with Jira, Confluence, Git Hub, and Agile delivery methodologies.
- Strong analytical, troubleshooting, and problem-solving skills.
- Ability to identify response inconsistencies, incorrect outputs, and potential AI hallucinations.
- Excellent communication, defect reporting, and test documentation skills.
- Experience developing automated validation frameworks for AI-generated responses.
- Familiarity with LLM evaluation techniques, conversational testing, and regression testing for AI applications.
- Experience monitoring chatbot quality in production environments.
- Understanding of test data management and quality metrics for AI-enabled systems.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).