Health AI Conversation Evaluator

About the job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark
, General Catalyst
, Peter Thiel
, Adam D'Angelo
, Larry Summers
, and Jack Dorsey
.

Position: Generalist Annotator — Health AI Conversation Quality Evaluation
Type:
Contract
Compensation: $20–$160/hour
Location:
Remote

• Evaluate the relevance and clarity of AI responses
  . Ensure the AI answers the question in understandable language for non-experts.
• Detect sycophancy in AI interactions. Identify when the AI tells users what they want to hear instead of what they need.
• Assess the tone and outcome of conversations. Determine if the user leaves informed and calm or anxious and uninformed.
• Provide written justification for ratings below the top option. Explain what went wrong and suggest improvements.
• Work independently to review conversations ranging from 1 to 7 turns. Spend 2–13 minutes per conversation based on length.

• Strong written English skills.
• Ability to explain judgments in two to four specific sentences.
• Careful reading to notice omissions in responses.
• Comfort with applying a detailed rubric consistently.
• Reliable availability for time-boxed batches.

• Experience in annotation, human-feedback, evaluation, or content review.
• Background in explaining technical concepts to non-experts.
• Experience with rubric-based grading or quality assurance.

• Using an AI tool to write comments is prohibited. This work captures human judgment that AI lacks. AI-written feedback corrupts the dataset and is checked for.