Data Science QA Lead

About Open Train

Open Train AI is the hiring and contracting organization for this role. Open Train is the #1 platform for finding and building careers in AI training and data labeling, helping contributors discover projects, build a professional profile, and apply to opportunities in minutes.

Creating an Open Train account is free, and this opportunity offers a way to contribute directly to the development and evaluation of modern AI systems.

AI training is the human side of building artificial intelligence. People review examples, evaluate model outputs, check technical accuracy, and provide feedback that helps AI systems become more useful and reliable.

In this role, your data science expertise will support evaluation of AI-generated explanations, analytical workflows, code, experiments, and conclusions. The work is remote and can be a flexible way to apply specialized skills to cutting-edge AI projects.

Open Train AI is recruiting a Data Science QA Lead to review AI-generated data science content and trainer QA work. You will assess statistical accuracy, model selection, code correctness, reproducibility, metric interpretation, business context, instruction following, and rubric adherence.

You will also provide precise written feedback, identify recurring quality issues, help update trainers and QAs, support contributor onboarding, maintain quality documentation, and improve QA processes. This is a remote hourly contractor role for US-based contributors working 20 or more hours per week.

• Advertised rate of up to $110 per hour
• Part-time contractor engagement
• Remote work available in the United States
• English-language role requiring strong written communication

You will evaluate both the technical substance and communication quality of data science work. Your reviews will help ensure that AI-generated content is accurate, reproducible, methodologically sound, and aligned with project requirements.

• Review AI-generated data science explanations, Python, R, and SQL snippets, modeling workflows, dashboards, experiment designs, and step-by-step reasoning.
• Check for data leakage, flawed assumptions, incorrect metrics, weak methodology, non-reproducible code, and misleading conclusions.
• Assess analytical work against rubrics covering statistical accuracy, model selection, code correctness, business context, instruction following, and rubric adherence.
• Communicate guideline changes and workflow updates to trainers and QAs.
• Create and maintain style guides, trackers, FAQs, examples, honeypots, calibration tasks, and onboarding materials.
• Support onboarding and training calls for contributors.
• Identify recurring issues and help improve QA processes.

The role requires a strong quantitative background and the ability to review analytical work against detailed rubrics. Candidates should be comfortable explaining technical findings clearly in written English and coordinating with distributed contributors.

• Degree in data science, statistics, computer science, machine learning, mathematics, economics, engineering, or a related quantitative field.
• Strong English communication skills for clear technical feedback and team coordination.
• At least 3 years of experience in data science, analytics, machine learning, statistical modeling, experimentation, data engineering, technical review, or data science education.
• Strong understanding of statistics, model evaluation, experimentation, regression, classification, clustering, and validation methods.
• Familiarity with Python, pandas, Num Py, scikit-learn, SQL, Jupyter, matplotlib, R, Spark, Git, MLflow, notebooks, dashboards, and cloud or data platforms.
• Experience with AI training, data annotation, LLM evaluation, data science QA, or rubric-based technical review is a strong plus.

Experience supporting distributed teams and maintaining clear quality resources will help you succeed. The work involves coordinating updates, organizing documentation, and keeping review standards consistent across projects.

• Experience leading or supporting remote teams of trainers, annotators, analysts, data scientists, engineers, educators, or QAs.
• Comfort using Discord, Google Sheets, Google Docs, trackers, dashboards, Git Hub, and project management systems.
• Strong organizational skills and the ability to maintain documentation and quality resources.
• Availability for 20 or more hours per week.