Generalist Expert AI Evaluator

Evaluate AI-generated responses remotely for clarity, accuracy, and human reasoning while developing prompts across generalist subject areas. Work asynchronously as an independent contractor.
Mercor connects creative and technical talent with leading AI research labs and is seeking a Generalist Expert. The assignment lasts approximately two weeks, with potential extension, and requires ten to twenty-five hours weekly. You will benchmark model responses, identify failure points, and maintain consistent quality across summaries, Q&A, and synthesis tasks. Research, writing, editing, education, or related analytical experience supports this work.

Key Responsibilities:

Review and revise responses for clarity and accuracy
Create prompts spanning a range of generalist topics
Benchmark responses by identifying model failure points
Compare variations in length, style, and system prompts
Meet deadlines while improving AI model performance
Requirements:
Two or more years of relevant analytical experience
Strong ability to explain complex topics concisely
Careful attention to detail and critical thinking
Comfort working independently as guidelines evolve
Familiarity with LLMs preferred
Bring research judgment, concise communication, and response evaluation skills to Mercor’s AI benchmarking and prompt-development work.