Skip to content
×

Sr. Legal AI Prompter, Evaluator

Listing for: USA018 Thomson Reuters Holdings Inc.
Full Time position Listed on 2026-11-13
Job specializations:
Salary/Wage Range or Industry Benchmark: 120000 - 222000 USD Yearly USD 120000.00 222000.00 YEAR

Job Description & How to Apply Below

Sr. Legal AI Prompter / Evaluator Job Description

Sr. Legal AI Prompters / Evaluators are responsible for improving CoCounsel Legal by writing and refining prompts, skills, and agent configurations; experimenting with large language models (LLMs), including Thomson, our proprietary LLM; and evaluating the results. They partner closely with specialized legal subject matter experts (SMEs), software engineers, and data scientists to make sure our products deliver accurate, legally sound outputs that legal professionals can trust.

As senior members of the team, they take a leading role in the hands‑on development of LLM‑driven solutions, design and run experiments to find out what works, and help raise the team’s capabilities. The ideal candidate is an experienced, practiced lawyer with a passion for technology and a strong understanding of emerging technologies in the legal profession, who can think critically about AI risks and opportunities and translate legal expertise into practical, high‑quality AI outputs.

Key Responsibilities
  • Create, test, and refine prompts, skills, and agent configurations for CoCounsel Legal to ensure accurate, legally sound, and contextually appropriate responses
  • Experiment with Thomson and other LLMs on legal tasks, comparing results to inform which model best fits each use case
  • Build gold‑standard datasets with SMEs, coordinate evaluations, and design evaluation schemes, including rubrics, metrics, success criteria, and LLM‑as‑judge prompts, to test model outputs and inform launch readiness decisions
  • Write and run scripts, experiments, and automated tests that ensure quality and consistency over large datasets and time periods
  • Monitor production performance, analyze failure modes and identify where in the pipeline they originate (prompting, tooling, retrieval, or content), and provide strategic feedback to software engineers and data scientists
  • Collaborate with software engineers, data scientists, and cross‑functional teams as they integrate LLMs into multi‑step systems, and evaluate the feasibility and potential impact of new product ideas
  • Engage with legal professionals, industry leaders, and internal stakeholders to gather insights and validate product concepts, and stay informed about developments in legal technology and LLMs
  • Mentor colleagues and draft guidance materials, reusable prompts, and best‑practice documentation on getting the most out of LLMs
Required Qualifications
  • Juris Doctor degree or equivalent law degree and bar admission in a US or Canadian jurisdiction
  • 5-7 years of combined experience, including at least 2-3 years of legal practice, with the remaining years in related work such as collaborating with software engineers, building or testing products, applying generative AI, or writing code
  • Proven expertise applying generative AI, including prompt engineering and evaluating model outputs, and an understanding of how LLMs work and where they fail
  • Proficiency working in a Python code base and using Git, including writing, modifying, and debugging Python scripts, calling LLMs through code (e.g., OpenAI, Anthropic, Gemini), and using coding agents to build and iterate on that work
  • Excellent communication and collaboration skills, with the ability to work effectively with legal SMEs and across functions
Preferred Qualifications
  • Have substantial experience working with and testing legal products and technology, including online legal research and practice tools
  • Have hands‑on experience designing evaluations, such as rubrics, human grading frameworks, or LLM‑as‑judge prompts
  • Are familiar with agentic and RAG systems, including how retrieval, tool use, and prompting each affect output quality
  • Are a self‑starter who is comfortable with ambiguity and learns new…
To View & Apply for jobs on this site that accept applications from your location or country, tap the button below to make a Search.
(If this job is in fact in your jurisdiction, then you may be using a Proxy or VPN to access this site, and to progress further, you should change your connectivity to another mobile device or PC).
View / Apply for Jobs
Matching My Jurisdiction

Search for further Jobs Here:

SEARCH JOBS: 10+ Million Listings
(Try combinations for better Results! Or enter less keywords for broader Results)
Education Level
Experience Level (years)
Posted in last:
Salary