Technical AI Evaluation Analyst; LATAM
Listed on 2026-09-29
-
Quality Assurance - QA/QC
Quality Control - QC Analysts/Managers, IT QA Tester / Automation, QA Specialist - Analyst/Manager
About Gramian
Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong background in software engineering and leadership, we help companies build high-performing teams by matching them with professionals who truly fit their needs.
About the RoleWe are seeking a detail-oriented AI Evaluation & Quality Assurance Specialist to review the quality, correctness, and fairness of tasks designed to evaluate AI agents. You will inspect reference solutions, grading logic, execution traces, and generated deliverables to identify task defects, evaluation errors, and unjustified model failures. This role requires strong technical fluency, independent analytical judgment, and the ability to produce clear, evidence-based feedback.
LOCATION: Remote – Latin America (LATAM)
CONTRACT: Hourly Contractor
COMMITMENT: 40 hours per week
TIME OVERLAP: 8 hours of mandatory PST overlap
DURATION: 10 weeks
START DATE: Immediately
Key Responsibilities- Validate task quality by checking instructions, source materials, reference solutions, and evaluation criteria for consistency and completeness.
- Review AI agent execution traces, tool calls, and deliverables to determine whether outcomes are justified.
- Audit grading logic to identify brittle checks, incorrect expected answers, unsupported rubric criteria, and unfair penalties for valid alternative solutions.
- Investigate discrepancies between model performance, grader results, and expected outcomes.
- Distinguish genuine model limitations from task defects, grader errors, and environment or tool failures.
- Independently assess automated QC findings rather than accepting them without verification.
- Document concise, evidence-backed findings and provide actionable, reproducible feedback.
- Flag uncertainty and verify that implemented revisions resolve previously identified issues.
- Comfortable reading and interpreting Python, SQL, shell scripts, structured data, and execution logs.
- Experience reviewing technical workflows, software behavior, data outputs, or evaluation logic.
- Strong analytical skills, including the ability to verify calculations and reconcile conflicting evidence.
- Demonstrated ability to assess the correctness and completeness of technical deliverables.
- Strong written English with experience providing clear, specific, and reproducible feedback.
- High attention to detail when identifying inconsistencies, missing information, and evaluation defects.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).