ESRC AI Project Post-Doctoral Researcher upon Tyne

Job Description

ABOUT THE ROLE

As a Research Associate, you will take a leading role in the project Trustworthy AI for Peer Review:
Developing and Validating Composite Research Quality Indicators. Your work will focus on constructing high-quality
-aligned corpora of research outputs, using and extending the full-text cache and metadata pipelines developed through our software:
Macroscope. You will collect, clean, link, and document text and bibliographic data so that it can support rigorous testing of AI-assisted research assessment.

You will be responsible for developing reproducible data workflows that link REF
2021 submissions, Units of Assessment, outcome profiles, institutional metadata, bibliographic records, abstracts, and, where licensing permits, full-text content. This will involve working with sources such as Open Alex, Crossref, Unpaywall, Scopus and , designing robust extraction and matching methods, maintaining provenance and quality-control records, and producing well-structured datasets for downstream analysis.

The role will also contribute to the projects LLM-based evaluation work, helping to design, run, and analyse experiments that test how large language models perform in
-style assessment tasks. This will include supporting prompt and model comparison, repeated-run evaluation, assessment of stability and disagreement, and analysis of potential biases or confounding effects. The post provides an opportunity to work at the frontier of trustworthy AI, research evaluation, scientometrics, and large-scale text infrastructure, with scope to publish, present, and help shape policy-relevant demonstrators.

The successful candidate will also have the opportunity to support in the development of Semantic Space, an Academic Intelligence company that supports strategic decision making for research intensive organisations.

The role is fixed term for 12 months .

ABOUT THE PROJECT

Trustworthy AI for Peer Review:
Developing and Validating Composite Research Quality Indicators is a UKRI/ESRC Metascience project investigating how artificial intelligence can support high-stakes research assessment without displacing expert judgement. Research assessment through peer review, journal editorial processes and the Research Excellence Framework () is essential to the research system, but it is increasingly costly, time-consuming and difficult to scale. At the same time, large language models are creating new possibilities for analysing and evaluating research outputs.

The project addresses a central challenge: AI-derived research indicators are often presented as single scores, with limited visibility of uncertainty, disagreement, instability or bias. The project will develop and validate new approaches that make the reliability of AI-augmented assessment explicit and usable by decision-makers. It will combine benchmark datasets, LLM evaluation, scientometric indicators, uncertainty-aware modelling, Trust Cards, and user-centred demonstrators for s, journal editors, publishers and research managers.

Key research areas include:

• Building FAIR, multi-resolution benchmark datasets linking s, journal peer-review data, books, bibliographic metadata, abstracts and, where appropriate, full-text corpora.
• Developing and testing LLM-based approaches to
  -style and peer-review assessment, including repeated-run evaluation, prompt comparison, model comparison, and calibration against known assessment outcomes.
• Creating indicators that communicate uncertainty, disagreement, stability, bias sensitivity and limits of valid use, rather than reducing complex judgements to opaque single scores.
• Developing transparent governance and communication tools, including Trust Cards, open-source workflows, documentation and demonstrators for responsible use of AI in research evaluation.

The project is delivered by an interdisciplinary consortium led by Loughborough University with Northumbria University, Durham University, Heriot-Watt University,…