Reducing research bureaucracy in UK higher education: Can generative AI assist with the internal evaluation of quality?

TL;DR

This study explores how generative AI, specifically ChatGPT, can assist UK higher education institutions in streamlining research quality evaluations, potentially reducing bureaucracy and resource costs while maintaining assessment accuracy.

Contribution

The paper introduces an experimental methodology using ChatGPT to score research papers, demonstrating AI's potential to support and streamline internal research evaluation processes.

Findings

AI scoring boundaries aligned with REF outcomes

AI can identify borderline cases for human review

Potential for reducing resource burden in research assessment

Abstract

This paper examines the potential for generative artificial intelligence (GenAI) to assist with internal review processes for research quality evaluations in UK higher education and particularly in preparation for the Research Excellence Framework (REF). Using the lens of function substitution in the Viable Systems Model, we present an experimental methodology using ChatGPT to score and rank business and management papers from REF 2021 submissions, "reverse engineering" the assessment by comparing AI-generated scores with known institutional results. Through rigourous testing of 822 papers across 11 institutions, we established scoring boundaries that aligned with reported REF outcomes: 49% between 1* and 2*, 59% between 2* and 3*, and 69% between 3* and 4*. The results demonstrate that AI can provide consistent evaluations that help identify borderline evaluation cases requiring additional human scrutiny while reducing the substantial resource burden of traditional internal review processes. We argue for application through a nuanced hybrid approach that maintains academic integrity while addressing the multi-million pound costs associated with research evaluation bureaucracy. While acknowledging these limitations including potential AI biases, the research presents a promising framework for more efficient, consistent evaluations that could transform current approaches to research assessment.