The disruption index suffers from citation inflation and is confounded by shifts in scholarly citation practice

TL;DR

This paper reveals that the widely used disruption index (CD) is systematically biased by citation inflation and changing citation practices, which confound its ability to accurately measure innovation over time.

Contribution

The study identifies and demonstrates how citation inflation and behavioral shifts distort the disruption index, proposing a more accurate understanding of research disruptiveness.

Findings

CD decreases over time due to citation inflation and behavioral factors.

Citation inflation is caused by increasing reference list lengths in publications.

Disruptiveness trends vary with team size, shifting from negative to positive for larger teams.

Abstract

Measuring the rate of innovation in academia and industry is fundamental to monitoring the efficiency and competitiveness of the knowledge economy. To this end, a disruption index (CD) was recently developed and applied to publication and patent citation networks (Wu et al., Nature 2019; Park et al., Nature 2023). Here we show that CD systematically decreases over time due to secular growth in research and patent production, following two distinct mechanisms unrelated to innovation -- one behavioral and the other structural. Whereas the behavioral explanation reflects shifts associated with techno-social factors (e.g. self-citation practices), the structural explanation follows from `citation inflation' (CI), an inextricable feature of real citation networks attributable to increasing reference list lengths, which causes CD to systematically decrease. We demonstrate this causal link by way of mathematical deduction, computational simulation, multi-variate regression, and quasi-experimental comparison of the disruptiveness of PNAS versus PNAS Plus articles, which differ only in their lengths. Accordingly, we analyze CD data available in the SciSciNet database and find that disruptiveness incrementally increased from 2005-2015, and that the negative relationship between disruption and team-size is remarkably small in overall magnitude effect size, and shifts from negative to positive for team size $\geq$ 8 coauthors.