Assistant Research Scientist; PREP

Overview

PREP Research Associate

Project

Description:

This project focuses on using Large Language Models (LLMs) to provide annotations of evaluation data (LLM as judge) and the design of an Inter-Annotator Agreement study to assess the reliability of both human and LLM annotations. The candidate will explore indicators of AI-related risk, learn how to identify them, and provide annotators with examples to annotate the presence of various risks.

The project aims to develop an annotation framework for AI risk assessment and establish metrics for data quality in AI risk research, supporting broader work at NIST in assessing and measuring the validity and reliability of AI-related risks in data annotation.

Research

Title:

Reliability of Human and LLM Annotations for AI Risk Assessment

• Gain familiarity with existing literature on data annotation and LLM as judge
• Understand NIST's role and ongoing efforts in assessing and measuring the validity and reliability of AI-related risks in data annotation
• Contribute to developing an annotation framework for AI risk assessment
• Collaborate effectively with cross-functional and interdisciplinary stakeholders to ensure successful project outcomes

Contributions to a NIST report that supports ongoing NIST AI evaluation efforts focused on the design of an Inter-Annotator Agreement to assess the reliability of both human and LLM annotations.

• Background in Computer Science, Data Science, or related field
• Undergraduate or graduate student education level
• Strong interest in data annotation and AI risks
• Familiarity with scientific reading and technical writing

Number of Positions: 1

The position is expected to be primarily telework-based, with some in-person meetings at NIST's Gaithersburg campus as needed. Average Hours Per Week: 20

Please upload the following with your application: CV/Resume, Self portraits, Phone number, Home address/Country, Citizenship status, Languages spoken, Sex/Gender. Privacy and compliance information follows per institutional policy.

Privacy Act Statement - Authority and Purpose information related to the PREP program and applications.

The PREP program details, routine uses, and disclosure information are described in the original posting. Furnishing information is voluntary. By submitting, you consent to use for program purposes.

Equal Opportunity Employer - The Johns Hopkins University does not discriminate on the basis of sex, gender, marital status, pregnancy, race, color, ethnicity, national origin, age, disability, religion, sexual orientation, gender identity or expression, veteran status or other legally protected characteristic.

Pre-Employment Information - For special accommodations during the pre-employment process, contact the HR office. For more information about accessibility, visit the university accessibility site.

Background Checks - A pre-employment background check including education verification may be required.

EEO is the Law - See the university’s EEOC rights notice for more information.

Vaccine Requirements - Vaccination guidance and exemptions as applicable by campus and program, with specifics for School of Medicine as described in the original posting.

The following provisions may apply depending on campus. Your recruiter will advise accordingly.