AI Evaluation Engineer

Elevate customer experience with Dialpad as an AI Evaluation Engineer in Kitchener, Canada. Own evaluation strategies for our advanced AI systems while collaborating with cross-functional teams.

In this role, you'll join the AI Evaluation team, focusing on validating agentic AI systems and enhancing model evaluation methods. Your responsibilities will encompass developing LLM-judge metrics, curating datasets, and conducting structured error analyses to ensure product readiness. Strong analytical skills and the ability to work with complex AI technologies are essential for success.

Key Responsibilities:

• Design and execute validation strategies across workflows

• Build and enhance regression evaluations and A/B tests

• Co-own LLM-judge metric development and calibration

• Create and manage data annotation jobs for evaluations

• Investigate bugs and determine escalation or analysis needs

Requirements:

• Bachelor’s or Master's degree in a related field

• 3+ years in QA or model evaluation for AI products

• Experience with structured test strategies and AI systems

• Strong analytical skills for quality pattern identification

• Effective communication with cross-functional teams

Contribute to innovative AI-driven solutions as an AI Evaluation Engineer at Dialpad in Kitchener.
#J-18808-Ljbffr