×
Register Here to Apply for Jobs or Post Jobs. X

AI Agent Evaluation Engineer

Job in Moncton, New Brunswick, Canada
Listing for: Outlier
Full Time position
Listed on 2026-11-04
Job specializations:
  • Software Development
    AI Engineer (Applied/Software), AI QA / Validation Engineer, Software Testing
Job Description & How to Apply Below
Evaluate autonomous AI agents through rubric writing, trace debugging, and stress testing while working remotely as a freelance contributor for Outlier.
Outlier helps innovative companies improve AI agents by collecting human feedback. This work focuses on training large language models to function as proactive, multi-step systems. You will assess how agents design, coordinate, and optimize complex workflows. Applicants need at least two years of relevant experience.

Key Responsibilities:

Debug agent traces across multi-turn interactions
Write detailed evaluation rubrics
Stress test autonomous agent behavior
Assess complex system behaviors with precision
Support training for generative AI systems
Requirements:

Experience with backend engineering, AI automation, or systems integration
Ability to maintain production-grade modular software
Practical experience with live environments
Clear, high-density technical feedback skills
Experience handling persistent state or session discovery preferred
Use technical feedback, agent testing, and systems integration experience to strengthen autonomous AI evaluation projects with Outlier.
To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
 
 
 
Search for further Jobs Here:
(Try combinations for better Results! Or enter less keywords for broader Results)
Location
Increase/decrease your Search Radius (miles)
0
200
Filters
Education Level
Experience Level (years)
Posted in last:
Salary