AI Evaluation Engineer

AI Evaluation Engineer

Locations:
Charlotte, North Carolina, United States;
Denver, Colorado, United States;
New York, New York, United States

Judi Health is an enterprise health technology company providing a comprehensive suite of solutions for employers and health plans, including:

• Judi Rx, a public benefit corporation delivering full-service pharmacy benefit management (PBM) solutions to self-insured employers,
• Judi Health™, which offers full-service health benefit management solutions to employers, TPAs, and health plans, and
• Judi®, the industry's leading proprietary Enterprise Health Platform (EHP), which consolidates all claim administration-related workflows in one scalable, secure platform.

Together with our clients, we're rebuilding trust in healthcare in the U.S. and deploying the infrastructure we need for the care we deserve. To learn more, visit (Use the "Apply for this Job" box below)..

As an AI Evaluation Engineer at Judi Health, you will build the testing frameworks, metrics, and tooling used to assess the safety, reliability, and accuracy of AI models and autonomous agents in production. This role bridges the gap between model development and real-world usage by translating ambiguous product goals into measurable quality targets.

We're looking for someone to lead evaluation end-to-end — from unit and integration testing to offline, online, and statistical evaluations of probabilistic systems. What we need is someone who can design and operate robust evaluation frameworks, partner with scientists and engineers, and ensure we can confidently answer questions like: "Did this change improve or degrade quality, safety, or user outcomes?"

• Build data evaluation pipelines that collect production conversations and agent interactions
• Reconstruct full sessions from traces, logs, recordings, and transcripts
• Apply labeling and scoring using human feedback signals (surveys, sentiment, outcomes) and automated evaluators (e.g., LLM-as-judge)

• Own weekly and on-demand automated evaluation runs against staging and production
• Define benchmarks that track accuracy, reliability, and safety-related signals
• Produce trend dashboards that clearly answer: "Did this deploy change quality or risk?"

• Design and extend a standardized evaluation framework that supports multiple agent types and workflows
• Translate high-level product expectations into concrete success criteria and metrics
• Ensure new agents and features can be evaluated consistently with minimal friction

• Build APIs and internal tools so data scientists and engineers can go from "interesting scenario" to "included in the eval suite" quickly
• Enable scenario curation, dataset management, and eval execution without deep infrastructure knowledge

• Provide shared visibility into prompt, model, and agent experiments
• Enable reproducibility and comparison across runs so teams can build on each other's work instead of operating in silos

• Build and maintain ETL pipelines for heterogeneous data sources (traces, logs, transcripts, user feedback)
• Implement complex data stitching and session reconstruction logic
• Manage dataset versioning, provenance, and lifecycle

• Develop dashboards and monitoring tools for AI quality metrics
• Integrate evaluations into CI/CD pipelines for scheduled and gated runs
• Implement alerting on quality and safety signals, not just infrastructure health

• Apply and extend LLM-as-judge evaluation patterns
• Design metrics and scoring approaches suitable for stochastic, non-deterministic systems
• Use tools like Lang Smith to track runs, traces, experiments, and evaluation results

• Partner closely with data science, engineering, and product teams
• Translate between research goals, product intent, and engineering constraints
• Help define what "good" looks like for AI behavior in production
• Advocate for strong developer experience and usability in the tools you build
• Responsible for adherence to…