More jobs:
Evaluation QA Lead; Agentic and RAG Evaluation
Job in
Boise, Ada County, Idaho, 83701, USA
Listing for:
Vettedlance
Part Time
position
Listed on 2026-11-08
Job specializations:
-
Quality Assurance - QA/QC
Data Annotation/ AI Labeling -
Research/Development
Data Annotation/ AI Labeling
Job Description & How to Apply Below
Experienced QA leads run quality for AI evaluation projects: rubric design, reviewer calibration and agreement tracking for agentic and RAG systems.
What you will do- Own QA for agentic-workflow and RAG/conversational evaluation projects.
- Write and refine rubrics and gold sets.
- Calibrate reviewers and track inter-annotator agreement (kappa, alpha).
- Audit samples and report quality trends.
- 4+ years owning QA on annotation, evaluation or software testing.
- Hands-on experience evaluating LLM, agent or RAG systems.
- Comfortable with agreement metrics and spreadsheets.
- Fluent written English.
- Remote, hourly contract (independent contractor)
- Part-time, flexible: 10-40 hrs/week
- Remote
Pay: $60 - $90 / hour (USD). Paid in USD, weekly. Rate set after a short paid assessment.
Are you ready to take on the Evaluation QA Lead (Agentic and RAG Evaluation) role at Vettedlance?
To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).