Skip to content
×

AI Evaluation Rubric Auditor

Remote / Online - Candidates ideally in
New York, New York City, Richmond County, New York, USA
Listing for: Confidential
Part Time, Remote/Work from Home position Listed on 2026-11-19
Job specializations:
Salary/Wage Range or Industry Benchmark: 60000 - 100000 USD Yearly USD 60000.00 100000.00 YEAR

Job Description & How to Apply Below

Location: New York

The work

We are testing AI agents on realistic shopping and delivery tasks. Your role as a Project Auditor is to audit the work done by our Project Reviewers. You will audit complete tasks (prompt and grading rubric) and determine if their rubric failures are fair:

  • Was the information available?
  • Does the rubric accurately grade what the prompt asked for?
  • Would a valid answer have been marked wrong?
  • Is the rubric structure correct per the guidelines?

This role includes providing feedback to reviewers regarding prompt and rubric issues.

Must-haves
  • Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.
  • Experience providing feedback to writers/reviewers
  • Consistent availability
  • Can read a long agent transcript and find where it went wrong.
  • Precise writing, and you check the data instead of trusting the reference answer.

This is a 100% remote contract role. Applicants must be located in:
Canada, United States, Argentina, Brazil, Chile, Colombia, Peru, United Kingdom.

To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction

Search for further Jobs Here:

SEARCH JOBS: 10+ Million Listings
(Try combinations for better Results! Or enter less keywords for broader Results)
Education Level
Experience Level (years)
Posted in last:
Salary