Metrics and peer review agreement at the institutional level

TL;DR

This study compares bibliometric indicators and peer review at the institutional level in Italy, finding that their agreement is comparable to peer reviewer agreement, suggesting metrics could substitute peer review in some contexts.

Contribution

It provides a hierarchical Bayesian analysis of agreement between metrics and peer review at the institutional level, highlighting potential for metrics to replace peer review in research assessments.

Findings

Higher agreement at institutional than publication level

Metrics and peer review agreement comparable to peer reviewer agreement

Potential for bibliometric indicators to substitute peer review in some fields

Abstract

In the past decades, many countries have started to fund academic institutions based on the evaluation of their scientific performance. In this context, post-publication peer review is often used to assess scientific performance. Bibliometric indicators have been suggested as an alternative to peer review. A recurrent question in this context is whether peer review and metrics tend to yield similar outcomes. In this paper, we study the agreement between bibliometric indicators and peer review based on a sample of publications submitted for evaluation to the national Italian research assessment exercise (2011--2014). In particular, we study the agreement between bibliometric indicators and peer review at a higher aggregation level, namely the institutional level. Additionally, we also quantify the internal agreement of peer review at the institutional level. We base our analysis on a hierarchical Bayesian model using cross-validation. We find that the level of agreement is generally higher at the institutional level than at the publication level. Overall, the agreement between metrics and peer review is on par with the internal agreement among two reviewers for certain fields of science in this particular context. This suggests that for some fields, bibliometric indicators may possibly be considered as an alternative to peer review for the Italian national research assessment exercise. Although results do not necessarily generalise to other contexts, it does raise the question whether similar findings would obtain for other research assessment exercises, such as in the United Kingdom.