Senior Software Engineer, LLM Evaluation

Help advance large language models by creating high-quality software engineering datasets, benchmarks, and evaluations alongside AI researchers. You will work across Python, JavaScript including ReactJS, C/C++, Java, Rust, and Go to develop reference solutions, improve code samples, and assess AI-generated output for real-world engineering quality.

• Curate code examples, build solutions, and correct code across Python, JavaScript including ReactJS, C/C++, Java, Rust, and Go for AI model training initiatives.
• Evaluate and refine AI-generated code for efficiency, scalability, and reliability.
• Partner with cross-functional teams to improve AI-driven coding solutions against industry performance benchmarks.
• Build agents that verify code quality and identify recurring error patterns.
• Develop and assess model capabilities across the software engineering lifecycle, including prototyping, architecture design, API design, production implementation, launch, experiments, monitoring, operations, and maintenance.
• Design automated verification mechanisms for solutions to software engineering tasks.

• Several years of software engineering experience, including at least 2 years of continuous full-time experience at a top-tier product company.
• Strong full-stack application development experience and the ability to deploy scalable, production-grade software with modern languages and tools.
• Deep knowledge of software architecture, design, development, debugging, and code-quality and code-review assessment.
• Excellent written and verbal communication skills, with the ability to provide clear, structured evaluation rationales.

Remote contract engagement.

Flexible schedule of at least 10 hours per week, with up to 40 hours per week available.

Some overlap with Pacific Time is required.

Independent contractor role, with no medical benefits or paid leave.