Big fish and small ponds: why the departmental h-index should not be used to rank universities

TL;DR

This paper critiques the use of the departmental h-index for university rankings, demonstrating its size dependence and proposing a simple correction method to improve its fairness and validity.

Contribution

It reveals the size bias in the collective h-index and proposes a straightforward adjustment to make it a more reliable metric for institutional assessment.

Findings

Random data reshuffling has little effect on the h-index, indicating size dependence.

The h-index reflects volume of research outputs more than quality.

A simple correction method can mitigate size bias in the h-index.

Abstract

The size-dependent nature of the so-called group or departmental h-index is reconsidered in this paper. While the influence of unit size on such collective measures was already demonstrated a decade ago, institutional ratings based on this metric can still be found and still impact on the reputations and funding of many research institutions. The aim of this paper is to demonstrate the fallacy of this approach to collective research-quality assessment in a simple way, focusing on the h-index in its original form. We show that randomly reshuffling real scientometric data (varying numbers of citations) amongst institutions of varying size, while maintaining the volume of their research outputs, has little effect on their departmental h-index. This suggests that the relative position in ratings based on the collective h-index is determined not only by quality (impact) of particular research outputs but by their volume. Therefore, the application of the collective h-index in its original form is disputable as a basement for comparison at aggregated levels such as to research groups, institutions or journals. We suggest a possible remedy for this failing which is implementable in a manner that is as simple and understandable as the h-index itself.