Common bibliometric approaches fail to assess correctly the number of important scientific advances for most countries and institutions

TL;DR

Most current bibliometric rankings fail to accurately measure scientific advances because they rely on percentile indicators that do not account for heterogeneity within research groups, leading to underestimations of true progress.

Contribution

The paper reveals limitations of percentile-based bibliometric indicators and proposes a method to improve accuracy by considering homogeneous research groups within countries and institutions.

Findings

Percentile indicators often underestimate scientific advances.

Homogeneity within research groups affects the validity of percentile-based rankings.

Japan's Nobel Prizes contradict its low percentile indicators, illustrating the method's limitations.

Abstract

Although not explicitly declared, most research rankings of countries and institutions are supposed to reveal their contribution to the advancement of knowledge. However, such advances are based on very highly cited publications with very low frequency, which can only very exceptionally be counted with statistical reliability. Percentile indicators enable calculations of the probability or frequency of such rare publications using counts of much more frequent publications; the general rule is that rankings based on the number of top 10% or 1% cited publications (Ptop 10%, Ptop 1%) will also be valid for the rare publications that push the boundaries of knowledge. Japan and its universities are exceptions, as their frequent Nobel Prizes contradicts their low Ptop 10% and Ptop 1%. We explain that this occurs because, in single research fields, the singularity of percentile indicators holds only for research groups that are homogeneous in their aims and efficiency. Correct calculations for ranking countries and institutions should add the results of their homogeneous groups, instead of considering all publications as a single set. Although based on Japan, our findings have a general character. Common predictions of scientific advances based on Ptop 10% might be severalfold lower than correct calculations.