Senior Software Engineer, AI Evaluation and Coding Agents

Help improve AI coding agents by reviewing their work across real-world software repositories. You will assess whether an agent understands a task, selects an appropriate approach, and produces code that is correct, robust, and maintainable. Your technical judgment and precise feedback will help create the evaluation signals used to strengthen model performance.

• Design, refine, and iterate on rubrics and evaluation criteria for preference data.
• Review and label data to a high quality standard, identifying subtle issues that may be missed by others.
• Build and maintain pipelines and infrastructure for data generation, collection, and evaluation workflows.
• Turn findings from data work into clear written updates, recommendations, and documentation for the team.
• Work closely with researchers and engineers to convert qualitative technical judgment into scalable processes.

• At least 5 years of hands-on software engineering experience.
• Strong proficiency in Python, Type Script or JavaScript, Go, or another major production language.
• Experience working in substantial, real-world codebases.
• Strong code-review skills and technical judgment.
• Ability to clearly explain why an implementation is correct, incorrect, or could be improved.
• Strong written communication skills.
• Experience using modern LLMs or AI coding tools.
• Experience with LLM evaluation, coding agents, RLHF, preference data, rubric design, or post-training is helpful but not required.

Remote independent-contractor engagement, available only to candidates located in North America.

Commitment of 20 to 40 hours per week, including at least 6 hours of overlap with Pacific Time each day.

Approximately 3 months in duration, with a start date as soon as possible.

Compensation $100 to $150 per hour.