Senior Software Development Engineer

About The Job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark
, General Catalyst
, Peter Thiel
, Adam D'Angelo
, Larry Summers
, and Jack Dorsey
.

Position: AI Model Evaluator
Type: Full-time or Part-time Contract Work
Compensation: $60–$100/hour
Location: Remote
Commitment: 20+ hours/week

• Evaluate LLM-generated responses to coding and software engineering queries for accuracy, reasoning, clarity, and completeness.
• Conduct fact‑checking using trusted public sources and authoritative references.
• Execute code and validate outputs using appropriate tools to ensure accuracy.
• Annotate model responses by identifying strengths, areas of improvement, and factual or conceptual inaccuracies.
• Assess code quality, readability, algorithmic soundness, and explanation quality.
• Ensure model responses align with expected conversational behavior and system guidelines.
• Apply consistent evaluation standards by following clear taxonomies, benchmarks, and detailed evaluation guidelines.

• BS, MS, or PhD in Computer Science or a closely related field.
• 5+ years of real‑world experience in software engineering or related technical roles.
• Expertise in at least two relevant programming languages (e.g., Python, Java, C++, JavaScript, etc.).
• Ability to solve Hacker Rank or Leet Code Medium and Hard–level problems independently.
• Experience contributing to well‑known open‑source projects, including merged pull requests.
• Significant experience using LLMs while coding and understanding their strengths and failure modes.
• Strong attention to detail and comfort in evaluating complex technical reasoning.

• Prior experience with RLHF, model evaluation, or data annotation work.
• Track record in competitive programming.
• Experience reviewing code in production environments.
• Familiarity with multiple programming paradigms or ecosystems.
• Experience explaining complex technical concepts to non‑expert audiences.

• Upload resume
• AI interview based on your resume
• Submit form

• For details about the interview process and platform information, please check:
• For any help or support, reach out to:

PS:
Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.