A critical comparative analysis of five world university rankings

TL;DR

This paper critically compares five major world university rankings, analyzing their methodologies, overlaps, and limitations to provide a nuanced understanding of their differences and implications.

Contribution

It offers a systematic comparison of five ranking systems, revealing their methodological differences and limitations in providing multi-faceted insights into university performance.

Findings

Current rankings are one-dimensional, offering isolated indicators rather than comprehensive data sets.

Differences in methodologies significantly affect institutional rankings.

Overlap and coverage vary widely among the five ranking systems.

Abstract

To provide users insight into the value and limits of world university rankings, a comparative analysis is conducted of 5 ranking systems: ARWU, Leiden, THE, QS and U-Multirank. It links these systems with one another at the level of individual institutions, and analyses the overlap in institutional coverage, geographical coverage, how indicators are calculated from raw data, the skewness of indicator distributions, and statistical correlations between indicators. Four secondary analyses are presented investigating national academic systems and selected pairs of indicators. It is argued that current systems are still one-dimensional in the sense that they provide finalized, seemingly unrelated indicator values rather than offering a data set and tools to observe patterns in multi-faceted data. By systematically comparing different systems, more insight is provided into how their institutional coverage, rating methods, the selection of indicators and their normalizations influence the ranking positions of given institutions.