Impact Factor volatility to a single paper: A comprehensive analysis

TL;DR

This study analyzes how individual highly-cited papers can significantly influence journal Impact Factors, revealing high volatility and potential biases that question the reliability of IF-based rankings.

Contribution

It provides a comprehensive analysis of the impact of single papers on journal Impact Factors across thousands of journals, highlighting the volatility and biases involved.

Findings

Top-cited papers can increase IF by over 0.5 points in many journals.

Small journals are more affected by highly-cited papers than large journals.

IF is highly sensitive to a few papers, questioning ranking reliability.

Abstract

We study how a single paper affects the Impact Factor (IF) by analyzing data from 3,088,511 papers published in 11639 journals in the 2017 Journal Citation Reports of Clarivate Analytics. We find that IFs are highly volatile. For example, the top-cited paper of 381 journals caused their IF to increase by more than 0.5 points, while for 818 journals the relative increase exceeded 25%. And one in 10 journals had their IF boosted by more than 50% by their top three cited papers. Because the single-paper effect on the IF is inversely proportional to journal size, small journals are rewarded much more strongly than large journals for a highly-cited paper, while they are penalized more for a low-cited paper, especially if their IF is high. This skewed reward mechanism incentivizes high-IF journals to stay small, to remain competitive in rankings. We discuss the implications for breakthrough papers appearing in prestigious journals. We question the reliability of IF rankings given the high IF sensitivity to a few papers for thousands of journals.