A discussion of measuring the top-1 percent most-highly cited publications: Quality and impact of Chinese papers

TL;DR

This paper examines the measurement of top-1 percent highly cited publications, highlighting how field normalization affects perceived quality and impact of Chinese scientific output compared to Western countries.

Contribution

It reveals that normalization methods significantly influence assessments of scientific quality, showing China’s strengths in social sciences and the US in biomedical fields.

Findings

China overtook the US in top 1 percent publications in 2019

Field normalization affects rankings of scientific quality

China outperforms the US in Business and Finance fields

Abstract

The top 1 percent most highly cited articles are watched closely as the vanguards of the sciences. Using Web of Science data, one can find that China had overtaken the USA in the relative participation in the top 1 percent in 2019, after outcompeting the EU on this indicator in 2015. However, this finding contrasts with repeated reports of Western agencies that the quality of Chinese output in science is lagging other advanced nations, even as it has caught up in numbers of articles. The difference between the results presented here and the previous results depends mainly upon field normalizations, which classify source journals by discipline. Average citation rates of these subsets are commonly used as a baseline so that one can compare among disciplines. However, the expected value of the top 1 percent of a sample of N papers is N 100, ceteris paribus. Using the average citation rates as expected values, errors are introduced by using the mean of highly skewed distributions and a specious precision in the delineations of the subsets. Classifications can be used for the decomposition, but not for the normalization. When the data is thus decomposed, the USA ranks ahead of China in biomedical fields such as virology. Although the number of papers is smaller, China outperforms the US in the field of Business and Finance in the Social Sciences Citation Index when p is less than .05. Using percentile ranks, subsets other than indexing based classifications can be tested for the statistical significance of differences among them.