Assistant Research Scientist; PREP

Overview

PREP Research Associate – National Institute of Standards and Technology (NIST) Professional Research Experience Program (PREP). Individuals in this position will perform technical work supporting the collaboration's scientific research.

Research

Title:

Reliability of Human and LLM Annotations for AI Risk Assessment.

The project focuses on using Large Language Models (LLMs) to provide annotations of evaluation data (LLM as judge) and designing an Inter‑Annotator Agreement study to assess the reliability of both human and LLM annotations. The candidate will explore indicators of AI‑related risk, develop an annotation framework, and establish metrics for data quality in AI risk research to support broader NIST efforts.

U.S. Citizen Preferred.

• Gain familiarity with existing literature on data annotation and LLM as judge.
• Understand NIST's role in assessing and measuring the validity and reliability of AI‑related risks in data annotation.
• Contribute to developing an annotation framework for AI risk assessment.
• Collaborate effectively with cross‑functional and interdisciplinary stakeholders to ensure successful project outcomes.

• Contributions to a NIST report supporting ongoing AI evaluation efforts focused on the design of an Inter‑Annotator Agreement to assess the reliability of human and LLM annotations.

• Background in Computer Science, Data Science, or related field.
• Bachelor's or graduate degree.
• Strong interest in data annotation and AI risks.
• Familiarity with scientific reading and technical writing.

EEO is the Law.