Assistant Research Scientist; PREP

PREP Research Associate

This position is part of the National Institute of Standards and Technology (NIST) Professional Research Experience Program (PREP). The PREP program involves staff from a range of backgrounds conducting scientific research across various fields. This role supports a collaboration in which we investigate the reliability of human and large language model annotations for AI risk assessment.

Reliability of Human and LLM Annotations for AI Risk Assessment

• Gain familiarity with existing literature on data annotation and the use of LLMs as judges.
• Understand NIST’s role and ongoing efforts in assessing the validity and reliability of AI‑related risks in data annotation.
• Contribute to developing an annotation framework for AI risk assessment.
• Collaborate effectively with cross‑functional and interdisciplinary stakeholders to ensure successful project outcomes.

• Contributions to a NIST report supporting ongoing AI evaluation efforts focusing on the design of an inter‑annotator agreement to assess the reliability of human and LLM annotations.

• Background in Computer Science, Data Science, or a related field.
• Education level:
  Bachelor’s degree or Graduate degree.
• Strong interest in data annotation and AI risks.
• Familiarity with scientific reading and technical writing.
• U.S. Citizen Preferred.

The referenced salary range represents the minimum and maximum salaries for this position and is based on Johns Hopkins University’s good faith belief at the time of posting. The actual compensation offered to the selected candidate may vary and will ultimately depend on multiple factors, including the successful candidate’s geographic location, skills, work experience, internal equity, market conditions, education/training and other factors, as reasonably determined by the University.

Johns Hopkins offers a total rewards package that supports our employees’ health, life, career and retirement. More information can be found here: /

The Johns Hopkins University is committed to equal opportunity for its faculty, staff, and students. The university does not discriminate on the basis of sex, gender, marital status, pregnancy, race, color, ethnicity, national origin, age, disability, religion, sexual orientation, gender identity or expression, veteran status or other legally protected characteristics. The university is committed to providing qualified individuals access to all academic and employment programs, benefits and activities on the basis of demonstrated ability, performance and merit without regard to personal factors or demographic characteristics that are irrelevant to the program involved.

For more information: https://(Use the "Apply for this Job" box below).-applicants