Can ChatGPT evaluate research environments? Evidence from REF2021

TL;DR

This study evaluates ChatGPT's ability to assess UK research environment statements for REF2021, finding strong correlations with expert scores but cautioning against full automation due to biases and inaccuracies.

Contribution

It demonstrates that LLMs like ChatGPT can support research environment evaluations with high correlation to experts, offering a potential tool to aid workload and consistency.

Findings

ChatGPT scores correlated positively with expert assessments in most UoAs.

ChatGPT-5 achieved the highest correlation (r=0.81) with REF scores.

Caution is advised due to potential biases and inaccuracies in LLM assessments.

Abstract

UK academic departments are evaluated partly on the statements that they write about the value of their research environments for the Research Excellence Framework (REF) periodic assessments. These statements mix qualitative narratives and quantitative data, typically requiring time-consuming and difficult expert judgements to assess. This article investigates whether Large Language Models (LLMs) can support the process or validate the results, using the UK REF2021 unit-level environment statements as a test case. Based on prompts mimicking the REF guidelines, ChatGPT 4o-mini scores correlated positively with expert scores in almost all 34 (field-based) Units of Assessment (UoAs). ChatGPT's scores had moderate to strong positive Spearman correlations with REF expert scores in 32 out of 34 UoAs: 14 UoAs above 0.7 and a further 13 between 0.6 and 0.7. Only two UoAs had weak or no significant associations (Classics and Clinical Medicine). From further tests for UoA34, multiple LLMs had significant positive correlations with REF2021 environment scores (all p < .001), with ChatGPT 5 performing best (r=0.81; $\rho$=0.82), followed by ChatGPT-4o-mini (r=0.68; $\rho$=0.67) and Gemini Flash 2.5 (r=0.67; $\rho$=0.69). If LLM-generated scores for environment statements are used in future to help reduce workload, support more consistent interpretation, and complement human review then caution must be exercised because of the potential for biases, inaccuracy in some cases, and unwanted systemic effects. Even the strong correlations found here seem unlikely to be judged close enough to expert scores to fully delegate the assessment task to LLMs.