What do we know about the disruption index in scientometrics? An overview of the literature

TL;DR

This paper reviews the disruption index in scientometrics, examining its variants, validity issues, biases, and performance, and provides guidelines for its appropriate use and future research directions.

Contribution

It offers a comprehensive overview of the disruption index literature, analyzing its technical properties, validity, biases, and comparing variants like DI1 and DI5.

Findings

DI5 shows higher convergent validity than DI1.

Disruption indices are affected by biases and data limitations.

More research is needed to validate disruption scores for research evaluation.

Abstract

The purpose of this paper is to provide a review of the literature on the original disruption index (DI1) and its variants in scientometrics. The DI1 has received much media attention and prompted a public debate about science policy implications, since a study published in Nature found that papers in all disciplines and patents are becoming less disruptive over time. This review explains in the first part the DI1 and its variants in detail by examining their technicaland theoretical properties. The remaining parts of the review are devoted to studies that examine the validity and the limitations of the indices. Particular focus is placed on (1) possible biases that affect disruption indices (2) the convergent and predictive validity of disruption scores, and (3) the comparative performance of the DI1 and its variants. The review shows that, while the literature on convergent validity is not entirely conclusive, it is clear that some modified index variants, in particular DI5, show higher degrees of convergent validity than DI1. The literature draws attention to the fact that (some) disruption indices suffer from inconsistency, time-sensitive biases, and several data-induced biases. The limitations of disruption indices are highlighted and best practice guidelines are provided. The review encourages users of the index to inform about the variety of DI1 variants and to apply the most appropriate variant. More research on the validity of disruption scores as well as a more precise understanding of disruption as a theoretical construct is needed before the indices can be used in the research evaluation practice.