From rankings to funnel plots: the question of accounting for uncertainty when measuring university research performance

TL;DR

This paper applies funnel plot methodology to assess and visualize uncertainty in Italian universities' research performance, revealing many universities' indicators lack statistical significance for performance differences, raising questions about motivation.

Contribution

It introduces the use of funnel plots for university research performance measurement, accounting for uncertainty at discipline and university levels.

Findings

Most universities' indicators lack significant evidence of performance differences.

Uncertainty visualization can influence institutional motivation.

Funnel plots provide a statistically sound method for performance assessment.

Abstract

The work applies the funnel plot methodology to measure and visualize uncertainty in the research performance of Italian universities in the science disciplines. The performance assessment is carried out at both discipline and overall university level. The findings reveal that for most universities the citation-based indicator used gives insufficient statistical evidence to infer that their research productivity is inferior or superior to the average. This general observation is one that we could indeed expect in a higher education system that is essentially non-competitive. The question is whether the introduction of uncertainty in performance reporting, while technically sound, could weaken institutional motivation to work towards continuous improvement.