Data Scientist

Join to apply for the Data Scientist role at Mercor

We're seeking a data-driven analyst to conduct comprehensive failure analysis on AI agent performance across finance-sector tasks. You'll identify patterns, root causes, and systemic issues in our evaluation framework by analyzing task performance across multiple dimensions (task types, file types, criteria, etc.).

• Statistical Failure Analysis:
  Identify patterns in AI agent failures across task components (prompts, rubrics, templates, file types, tags).
• Root Cause Analysis:
  Determine whether failures stem from task design, rubric clarity, file complexity, or agent limitations.
• Dimension Analysis:
  Analyze performance variations across finance sub-domains, file types, and task categories.
• Reporting & Visualization:
  Create dashboards and reports highlighting failure clusters, edge cases, and improvement opportunities.
• Quality Framework:
  Recommend improvements to task design, rubric structure, and evaluation criteria based on statistical findings.
• Stakeholder Communication:
  Present insights to data labeling experts and technical teams.

• Statistical Expertise:
  Strong foundation in statistical analysis, hypothesis testing, and pattern recognition.
• Programming:
  Proficiency in Python (pandas, scipy, matplotlib/seaborn) or R for data analysis.
• Data Analysis:
  Experience with exploratory data analysis and creating actionable insights from complex datasets.
• AI/ML Familiarity:
  Understanding of LLM evaluation methods and quality metrics.
• Tools:
  Comfortable working with Excel, data visualization tools (Tableau/Looker), and SQL.

• Experience with AI/ML model evaluation or quality assurance.
• Background in finance or willingness to learn finance domain concepts.
• Experience with multi-dimensional failure analysis.
• Familiarity with benchmark datasets and evaluation frameworks.
• 2-4 years of relevant experience.