AI Evaluation Expert
Job in
Toronto, Ontario, C6A, Canada
Listed on 2026-10-22
Listing for:
Rootly
Full Time
position Listed on 2026-10-22
Job specializations:
-
IT/Tech
IT QA Tester / Automation, Data Annotation/ AI Labeling -
Quality Assurance - QA/QC
IT QA Tester / Automation, Data Annotation/ AI Labeling
Job Description & How to Apply Below
Join Rootly as an AI Evaluation Expert, where your insights will directly influence the future of incident management. This role focuses on rigorous testing and optimization of AI features.
As our AI Quality Engineer, you'll play a vital role at Rootly, requiring over five years in QA or AI/ML evaluation. Your mission is to design and execute tests, ensure quality, and collaborate closely with teams to refine AI features. This position values individuals with sharp analytical capabilities and effective communication skills.
Key Responsibilities:
• Execute prompt-based tests covering various inputs
• Analyze AI behavior for relevance and accuracy
• Create frameworks for ongoing AI performance evaluation
• Document findings on AI behavior and improvements
• Define quality metrics for stakeholder reports
Requirements:
• 5+ years in QA, AI, or related roles
• Expertise with LLM and agentic AI systems
• Excellent prompt engineering instincts
• Comfortable with evaluation tools and scripts
• Strong communicator for technical findings
Shape the future of incident management with your evaluation expertise at Rootly.
To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
Search for further Jobs Here:
×