Specification uncertainty: What the disruption index tells us about the (hidden) multiverse of bibliometric indicators

TL;DR

This paper examines the variability and potential unreliability of the disruption index as a bibliometric indicator, emphasizing the need for transparency through multiverse analysis to ensure credible research outcomes.

Contribution

It highlights the hidden degrees of freedom in calculating the disruption index and advocates for multiverse-style methods to improve transparency and reliability.

Findings

Calculation of disruption scores involves numerous hidden degrees of freedom.

Analytical flexibility can threaten the credibility of bibliometric research.

Multiverse analysis can enhance transparency and trustworthiness.

Abstract

Following Funk and Owen-Smith (2017), Wu et al. (2019) proposed the disruption index (DI1) as a bibliometric indicator that measures disruptive and consolidating research. When we summarized the literature on the disruption index for our recently published review article (Leibel & Bornmann, 2024), we noticed that the calculation of disruption scores comes with numerous (hidden) degrees of freedom. In this Letter to the Editor, we explain based on the DI1 (as an example) why the analytical flexibility of bibliometric indicators potentially endangers the credibility of research and advertise the application of multiverse-style methods to increase the transparency of the research.