Use of the journal impact factor for assessing individual articles: Statistically flawed or not?

TL;DR

This paper critically examines statistical arguments against using journal impact factors for assessing individual articles, showing that such arguments are flawed and that impact factors can sometimes be appropriate indicators.

Contribution

The paper provides a theoretical and simulation-based analysis demonstrating that statistical criticisms of impact factors are unfounded and that impact factors can be valid under certain conditions.

Findings

Statistical arguments against impact factors are not supported by analysis.

Under some conditions, impact factors outperform citation counts as indicators.

The debate should focus on socio-technical implications rather than statistical flaws.

Abstract

Most scientometricians reject the use of the journal impact factor for assessing individual articles and their authors. The well-known San Francisco Declaration on Research Assessment also strongly objects against this way of using the impact factor. Arguments against the use of the impact factor at the level of individual articles are often based on statistical considerations. The skewness of journal citation distributions typically plays a central role in these arguments. We present a theoretical analysis of statistical arguments against the use of the impact factor at the level of individual articles. Our analysis shows that these arguments do not support the conclusion that the impact factor should not be used for assessing individual articles. Using computer simulations, we demonstrate that under certain conditions the number of citations an article has received is a more accurate indicator of the value of the article than the impact factor. However, under other conditions, the impact factor is a more accurate indicator. It is important to critically discuss the dominant role of the impact factor in research evaluations, but the discussion should not be based on misplaced statistical arguments. Instead, the primary focus should be on the socio-technical implications of the use of the impact factor.