Assessing the societal influence of academic research with ChatGPT: Impact case study evaluations

TL;DR

This study explores using ChatGPT to evaluate societal impact claims in academic impact case studies, finding it can support expert assessments with field-dependent accuracy, potentially streamlining societal impact evaluations.

Contribution

The paper demonstrates that ChatGPT can effectively support societal impact evaluations of academic research, with optimized input strategies improving correlation with expert judgments.

Findings

ChatGPT scores correlate positively with expert assessments across disciplines.

Inputting only the title and summary yields the best correlation.

Field-specific differences influence the effectiveness of ChatGPT evaluations.

Abstract

Academics and departments are sometimes judged by how their research has benefitted society. For example, the UK Research Excellence Framework (REF) assesses Impact Case Studies (ICS), which are five-page evidence-based claims of societal impacts. This study investigates whether ChatGPT can evaluate societal impact claims and therefore potentially support expert human assessors. For this, various parts of 6,220 public ICS from REF2021 were fed to ChatGPT 4o-mini along with the REF2021 evaluation guidelines, comparing the results with published departmental average ICS scores. The results suggest that the optimal strategy for high correlations with expert scores is to input the title and summary of an ICS but not the remaining text, and to modify the original REF guidelines to encourage a stricter evaluation. The scores generated by this approach correlated positively with departmental average scores in all 34 Units of Assessment (UoAs), with values between 0.18 (Economics and Econometrics) and 0.56 (Psychology, Psychiatry and Neuroscience). At the departmental level, the corresponding correlations were higher, reaching 0.71 for Sport and Exercise Sciences, Leisure and Tourism. Thus, ChatGPT-based ICS evaluations are simple and viable to support or cross-check expert judgments, although their value varies substantially between fields.