Can Recombination Displace Dominant Scientific Ideas

TL;DR

This study analyzes 41 million scientific articles to understand why the rate of breakthrough discoveries has not increased despite exponential growth in knowledge recombination potential, focusing on atypicality and disruption metrics.

Contribution

It introduces and empirically validates two distinct metrics, atypicality and disruption, to differentiate between types of scientific innovation and their roles in knowledge evolution.

Findings

Atypicality captures cross-topic recombination processes.

Disruption signifies replacement of dominant ideas within a topic.

Metrics reveal different pathways of scientific progress.

Abstract

The combination of diverse, pre-existing knowledge is a common explanation for scientific breakthroughs. However, a paradox exists: while scientific output and the potential for such recombination have grown exponentially, the rate of breakthrough discoveries has not. To explore this paradox, our study examines 41 million scientific articles from 1965 to 2024. We measure two key properties for each paper: atypicality, which quantifies the combination of knowledge from conceptually distant areas, and disruption. We demonstrate that these metrics capture distinct processes. Atypicality is characteristic of work that extends established concepts into new topical areas (a form of cross-topic recombination). Disruption, in contrast, signifies the replacement of a dominant idea within its own topic.