Caveats for the journal and field normalizations in the CWTS ("Leiden") evaluations of research performance

TL;DR

This paper critiques the Leiden normalization method for research performance evaluation, highlighting issues with averaging ratios in skewed distributions and proposing a more appropriate approach based on underlying data analysis.

Contribution

It demonstrates the flaws in the current Leiden normalization method and suggests a more statistically sound normalization procedure based on underlying data.

Findings

Leiden normalization can produce misleading results due to skewed distributions.

Averaging ratios is only valid for normal distributions, which is not the case here.

Using underlying data for normalization yields more accurate assessments.

Abstract

The Center for Science and Technology Studies at Leiden University advocates the use of specific normalizations for assessing research performance with reference to a world average. The Journal Citation Score (JCS) and Field Citation Score (FCS) are averaged for the research group or individual researcher under study, and then these values are used as denominators of the (mean) Citations per publication (CPP). Thus, this normalization is based on dividing two averages. This procedure only generates a legitimate indicator in the case of underlying normal distributions. Given the skewed distributions under study, one should average the observed versus expected values which are to be divided first for each publication. We show the effects of the Leiden normalization for a recent evaluation where we happened to have access to the underlying data.