Modeling within-department homogeneity in research quality rankings: an application to the Italian ISPD

TL;DR

This paper introduces a new statistical model to improve Italian academic department rankings by accounting for within-department score homogeneity, leading to fairer and more robust performance indices.

Contribution

It formalizes the impact of intra-department score homogeneity on rankings and proposes an adjusted index using a novel Betoidal distribution, validated with empirical data.

Findings

The adjusted index outperforms the original ISPD in empirical tests.

Model effectively accounts for intra-department correlation.

Simulation shows improved ranking fairness over existing methods.

Abstract

In this paper, we consider the academic department ranking system of Italy, which is based on a performance index named Indice Standardizzato di Performance Dipartimentale (ISPD). While critiques to the ISPD have been moved for its marked tendency to polarization, we here formalize a yet unexplored determinant of this phenomenon, that is, the presence of within-department homogeneity among the standardized scores used to build the index. We account for this intra-departmental correlation by modeling it as a function of departments' size. The proposed model, estimated via Maximum Likelihood, allows to build a fairer ranking procedure via the definition of a properly adjusted version of the ISPD. The estimation framework is also adapted to fit publicly available data, which are coarsened by rounding and/or left-truncated. To this end, a novel probability distribution termed Betoidal is introduced. Empirical evidence in favor of the proposed model is found in the 2017 and 2022 data. Moreover, a simulation study shows that the adjusted index significantly overcomes not only the original ISPD, but also other more data-demanding competing proposals.