×
Register Here to Apply for Jobs or Post Jobs. X

Autonomous Agent Evaluation Engineer

Job in Moncton, New Brunswick, Canada
Listing for: Outlier
Full Time position
Listed on 2026-11-04
Job specializations:
  • Software Development
    AI Engineer (Applied/Software), AI QA / Validation Engineer, Software Testing
Job Description & How to Apply Below
Evaluate autonomous agents through rubric writing, trace debugging, and stress testing. Work remotely with Outlier as a freelance contributor on flexible schedules.
Outlier helps innovative companies improve AI agents by providing human feedback for advanced generative systems. This work focuses on teaching large language models to coordinate multi-step, real-world architectural workflows. You will assess complex system behavior and deliver high-density technical feedback. The opportunity requires two or more years in backend engineering, AI automation, or complex systems integration.

Experience with modular production software, live environments, and multi-turn interactions is required. Applicants may apply from Australia, Canada, Finland, France, Italy, Norway, the United Kingdom, or the United States.

Key Responsibilities:

Develop evaluation rubrics for agent performance
Debug traces from multi-step agent workflows
Stress test autonomous agent behavior
Identify privacy leaks and authority escalation
Review indirect prompt injection failures
Requirements:
At least two years of relevant experience
Strong attention to complex system details
Clear, high-density technical feedback skills
Production-grade modular software experience
Live, multi-turn systems experience
Use evaluation, debugging, and systems-integration expertise to strengthen autonomous-agent performance for Outlier’s AI projects.
To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
 
 
 
Search for further Jobs Here:
(Try combinations for better Results! Or enter less keywords for broader Results)
Location
Increase/decrease your Search Radius (miles)
0
200
Filters
Education Level
Experience Level (years)
Posted in last:
Salary