AI Evaluation Specialist

Responsibilities

• Design and implement self-contained evaluation tasks, including prompts, supporting files, and detailed grading rubrics.
• Define clear, unambiguous written criteria to assess AI performance across diverse administrative and workflow scenarios.
• Meticulously observe and document AI agent behaviors, producing precise summaries and reports in high-quality English.
• Iterate and refine evaluation tasks and rubrics based on feedback to ensure robust benchmarking methodologies.
• Collaborate cross-functionally to adapt evaluation frameworks as project requirements evolve.

• Minimum 3 years of experience in roles emphasizing written precision and structured thinking (e.g., paralegal, technical writer, QA analyst, or research assistant).
• Native or fluent English writing skills with the ability to produce succinct and unambiguous observations.
• Proven skill in designing or applying rubric-based evaluation or structured scoring frameworks.
• High attention to detail and the ability to notice subtle patterns or inconsistencies.
• Proficiency with common SaaS tools, web browsers, file management, and document editing platforms.
• Strong self‑direction and the ability to work independently on ambiguous projects.

• Prior experience evaluating AI outputs or participating in technology‑driven process improvement projects.
• Background in developing or refining evaluation rubrics or scoring methodologies.
• Comfort working across multiple domains and adapting quickly to new workflow challenges.