A Global South Strategy for Evaluating Research Value with ChatGPT

TL;DR

This paper proposes a ChatGPT-based strategy for evaluating research relevance and quality tailored to the Global South, demonstrated through application to Mauritian research articles, highlighting its potential and limitations.

Contribution

It introduces a novel ChatGPT-driven research evaluation method that considers local context, addressing limitations of citation-based metrics in the Global South.

Findings

ChatGPT scores showed almost zero correlation with citation rates.

High-scoring articles often related to education, policy, or industry.

Articles mentioning the country or sectors had higher scores.

Abstract

Research evaluation is important for appointments, promotions, departmental assessments, and national science strategy monitoring. Whilst Global North universities often have sufficient senior researchers for effective peer review and enough trust in citation data to use it for supporting indicators, the same is less likely to be true in the Global South. Moreover, Global South research priorities may not align well with citation-based indicators. This article introduces a ChatGPT-based strategy designed to address both limitations, applying it to Mauritius. The strategy involves giving ChatGPT instructions about how to evaluate the quality of research from the perspective of a given Global South nation and then using it to score articles based on these criteria. Results from Mauritius show that ChatGPT's scores for 1,566 journal articles published between 2015 and 2021 have an almost zero correlation with both ChatGPT research quality scores and citation rates. A word association thematic analysis of articles with relatively high scores for value to Mauritius identified a range of plausible themes, including education, policy relevance, and industrial production. Higher scoring articles also tended to mention the country or an important commercial sector in the abstract. Whilst the evidence suggests that assessing the direct value to a country of journal articles using ChatGPT gives plausible results, this approach should be used cautiously because it has unknown accuracy and ignores the wider value of research contributions.