Normalization of peer-evaluation measures of group research quality across academic disciplines

TL;DR

This paper introduces a mathematical model to normalize peer-evaluation research quality measures across disciplines by accounting for group size effects, addressing flaws in current non-bibliometric assessment methods.

Contribution

It proposes a systematic normalization method based on a model that captures the relationship between research quality and group size, including critical mass effects.

Findings

Identifies two critical masses influencing research quality

Demonstrates the model's applicability across multiple disciplines

Suggests a normalization procedure aligning quality plateaus

Abstract

Peer-evaluation based measures of group research quality such as the UK's Research Assessment Exercise (RAE), which do not employ bibliometric analyses, cannot directly avail of such methods to normalize research impact across disciplines. This is seen as a conspicuous flaw of such exercises and calls have been made to find a remedy. Here a simple, systematic solution is proposed based upon a mathematical model for the relationship between research quality and group quantity. This model manifests both the Matthew effect and a phenomenon akin to the Ringelmann effect and reveals the existence of two critical masses for each academic discipline: a lower value, below which groups are vulnerable, and an upper value beyond which the dependency of quality on quantity reduces and plateaus appear when the critical masses are large. A possible normalization procedure is then to pitch these plateaus at similar levels. We examine the consequences of this procedure at RAE for a multitude of academic disciplines, corresponding to a range of critical masses.