AI Analytics Workflow Evaluator

About Open Train

Open Train is the #1 platform for finding and building careers in AI training and data labeling. Open Train AI is recruiting contractors for projects that help improve the systems behind modern AI assistants.

AI training is the human side of building artificial intelligence. Contributors review examples, test model behavior, evaluate responses, and identify errors so AI systems become more accurate, reliable, and useful.

This role focuses on evaluating AI assistants in realistic data-analysis workflows. Your analytics judgment will help determine whether generated SQL, figures, joins, filters, and time windows are accurate and logically sound.

• Remote work available worldwide
• Part-time contractor opportunity with 20+ hours per week
• Work directly with cutting-edge AI evaluation tasks
• Use practical analytics expertise to improve AI systems

As an AI Analytics Workflow Evaluator, you will assess how AI assistants perform data-analysis tasks connected to cloud data warehouses. The work combines hands-on SQL analysis, Snowflake environment management, product-behavior investigation, and careful evaluation of AI-generated results.

This opportunity is listed at an entry level, while advanced SQL and Snowflake capability are important for successful performance. At least three years of hands-on experience as a data analyst or analytics engineer is preferred.

• Payment: $50-$60 per hour
• Payment type: hourly
• Engagement: part-time contractor
• Expected availability: 20+ hours per week
• Working language:
  English

You will execute structured evaluation scenarios and compare AI-generated analytical outputs with source data. You will also maintain reproducible evaluation environments and document findings so scoring remains clear and consistent.

The work may include investigating undocumented product behavior, testing analytics connections, and participating in calibration sessions with other evaluators.

• Run scenarios involving anomaly investigation, KPI reporting, and data refreshes
• Write and execute SQL queries to verify AI-generated results against source data
• Audit figures, joins, filters, time windows, aggregations, and analytical logic
• Maintain, reset, and manage seeded Snowflake datasets
• Administer warehouse roles, permissions, grants, and access controls
• Configure and document connectivity and authentication across analytics and SaaS platforms
• Investigate undocumented product behavior and record findings clearly
• Participate in calibration sessions to support consistent grading and scoring

This role requires strong practical analytics judgment. You should be able to reconcile reported metrics with raw data, recognize subtle aggregation or logic errors, and explain your evaluation decisions clearly.

Experience administering database access and working with authentication or security integrations is valuable. Familiarity with business and finance analytics, including KPI definitions and reporting for leadership or external stakeholders, is helpful.

• Advanced SQL skills for validating AI-generated analytical results
• Snowflake expertise covering warehouses, query history, roles, grants, and access controls
• Ability to reconcile reported metrics with raw data
• Ability to identify aggregation, filtering, joining, or logic errors
• Strong written and verbal communication
• Careful documentation and sound judgment in rubric-based evaluation and QA
• Understanding of KPI definitions and business or finance reporting practices
• Experience using Claude or ChatGPT connectors for analytics workflows

A background as a data analyst or analytics engineer will help you contribute effectively, particularly if you have worked with cloud data warehouses and…