AI​/ML Evaluation Reviewer- Healthcare AI

Position: AI/ML Evaluation Reviewer- Healthcare AI (Contract)

About the Role

mpathic provides independent, multidisciplinary evaluations of healthcare AI technologies seeking deployment in real-world clinical settings. As part of our work supporting Utah's Office of Artificial Intelligence Policy (OAIP) regulatory sandbox, we are recruiting experienced machine learning (ML) and data science professionals to serve as technical reviewers on expert evaluation panels.

In this role, you will independently assess the technical evidence supporting healthcare AI applications, including model performance, validation methodology, generalizability, reliability, and proposed monitoring and safeguards. Your assessments will help inform recommendations regarding whether and under what conditions an AI technology may be appropriate for pilot deployment.

You will work alongside clinical and other subject‑matter experts to evaluate whether applicants’ technical claims are supported by credible evidence and whether their proposed evaluation and monitoring plans are appropriate for the intended healthcare setting and patient population.

This is an evaluation and scientific review role, not a model development position. We are looking for experts who can critically assess existing evidence, identify limitations, and communicate actionable recommendations to technical and non‑technical stakeholders.

What You'll Be Doing Responsibilities may include:

• Evaluate model performance and validation:
  Assess the appropriateness of validation methods, reference standards, sample sizes, statistical analyses, performance metrics, operating thresholds, and reported uncertainty.
• Assess data quality and representativeness:
  Review whether training, testing, and validation datasets are appropriate for the intended clinical application, including potential data leakage, subgroup performance differences, and generalizability across patient populations and healthcare settings.
• Evaluate technical reliability:
  Assess evidence of model robustness, reproducibility, known failure modes, and performance under atypical or out‑of‑distribution inputs.
• Review proposed evaluation plans:
  Determine whether applicants have established scientifically defensible methods, performance benchmarks, and success criteria for evaluating AI technologies during pilot deployment, particularly when pre‑deployment evidence is limited.
• Assess technical safeguards and human oversight:
  Evaluate whether proposed technical safeguards, escalation mechanisms, and human‑in‑the‑loop processes are operationally feasible and supported by appropriate testing.
• Evaluate monitoring and change management:
  Assess applicants’ plans for detecting performance degradation, monitoring model behavior, managing updates, and establishing criteria for intervention or rollback.
• Collaborate with multidisciplinary reviewers:
  Work alongside clinicians and other technical experts to identify concerns that require cross‑disciplinary judgment, particularly where model performance may affect clinical outcomes.
• Produce structured evaluation findings:
  Document strengths, limitations, missing evidence, and technical concerns using a standardized review framework. Provide clear, evidence‑based recommendations and identify what additional evidence or changes would resolve identified concerns.
• Participate in reviewer calibration and quality assurance:
  Contribute to discussions intended to improve consistency, transparency, and defensibility across evaluations.
• Maintain independence and confidentiality:
  Follow conflict‑of‑interest, confidentiality, and data‑handling requirements when reviewing proprietary or unpublished materials.

What We're Looking For

We are seeking technically rigorous, independent reviewers who can distinguish between a promising technology and…