Peer review and citation data in predicting university rankings, a large-scale analysis

TL;DR

This large-scale study demonstrates that citation-based indicators at the institutional level strongly correlate with peer review judgments, enabling more cost-effective university rankings with reduced reliance on peer review.

Contribution

The paper shows that citation indicators at the institutional level can reliably predict peer review outcomes, improving ranking accuracy and reducing evaluation costs.

Findings

High correlation (up to r=0.802) between citation indicators and peer review at institutional level

Reduced mean rank prediction error by 25% using citation data

Citation-based indicators can lower the need for extensive peer review in evaluations

Abstract

Most Performance-based Research Funding Systems (PRFS) draw on peer review and bibliometric indicators, two different methodologies which are sometimes combined. A common argument against the use of indicators in such research evaluation exercises is their low correlation at the article level with peer review judgments. In this study, we analyse 191,000 papers from 154 higher education institutes which were peer reviewed in a national research evaluation exercise. We combine these data with 6.95 million citations to the original papers. We show that when citation-based indicators are applied at the institutional or departmental level, rather than at the level of individual papers, surprisingly large correlations with peer review judgments can be observed, up to r <= 0.802, n = 37, p < 0.001 for some disciplines. In our evaluation of ranking prediction performance based on citation data, we show we can reduce the mean rank prediction error by 25% compared to previous work. This suggests that citation-based indicators are sufficiently aligned with peer review results at the institutional level to be used to lessen the overall burden of peer review on national evaluation exercises leading to considerable cost savings.