Evaluating research and researchers by the journal impact factor: is it better than coin flipping?

TL;DR

This paper critically examines the effectiveness of using journal impact factors to evaluate individual research papers, revealing that it often resembles random chance rather than a reliable metric.

Contribution

It quantifies the failure probability of using JIF for paper evaluation, showing its limitations compared to random guessing in typical scenarios.

Findings

Failure probability can approach 0.5 when JIFs are similar.

JIF-based evaluation often equates to coin flipping.

Large JIF differences reduce evaluation risk.

Abstract

The journal impact factor (JIF) is the average of the number of citations of the papers published in a journal, calculated according to a specific formula; it is extensively used for the evaluation of research and researchers. The method assumes that all papers in a journal have the same scientific merit, which is measured by the JIF of the publishing journal. This implies that the number of citations measures scientific merits but the JIF does not evaluate each individual paper by its own number of citations. Therefore, in the comparative evaluation of two papers, the use of the JIF implies a risk of failure, which occurs when a paper in the journal with the lower JIF is compared to another with fewer citations in the journal with the higher JIF. To quantify this risk of failure, this study calculates the failure probabilities, taking advantage of the lognormal distribution of citations. In two journals whose JIFs are ten-fold different, the failure probability is low. However, in most cases when two papers are compared, the JIFs of the journals are not so different. Then, the failure probability can be close to 0.5, which is equivalent to evaluating by coin flipping.