AI Evaluation Engineer

Position: AI Evaluation Engineer at Dialpad
Elevate customer experience at Dialpad as an AI Evaluation Engineer, focusing on agentic AI systems in Vancouver. Ensure quality and reliability across NLP and speech workflows while collaborating with cross-functional teams.

As a key player in our AI Evaluation team, you will co-own metric development and calibration, curate evaluation datasets, and lead structured error analysis. Your contributions will directly influence readiness decisions for our advanced AI voice and chat solutions. Engage in designing validation strategies and executing A/B comparisons to ensure optimal product performance.

Key Responsibilities:

• Design validation strategies for NLP and speech workflows

• Build and enhance regression evaluations and A/B tests

• Co-own LLM-judge metric development and calibration

• Create and monitor data annotation jobs for evaluations

• Investigate bugs and triage issues effectively

Requirements:

• Bachelor’s or Master’s in Computer Science or related field

• 3+ years in QA, test engineering, or model evaluation

• Experience with AI system test strategies

• Strong analytical skills for quality pattern identification

• Collaborative skills in technical documentation

Harness your skills in QA and AI validation to drive Dialpad's innovation forward.
#J-18808-Ljbffr