Innovation by Displacement

TL;DR

This paper challenges the traditional view that scientific breakthroughs mainly come from recombining existing ideas, showing instead that displacing dominant ideas often leads to more impactful innovations, with methods playing a key role.

Contribution

It introduces a new framework distinguishing between novelty and disruption, emphasizing the importance of idea displacement in scientific progress, supported by large-scale data analysis and interviews.

Findings

Disruptive ideas displace dominant ones within the same topic.

Novel ideas extend existing knowledge and attract immediate attention.

Methods more frequently drive disruptive breakthroughs than theories.

Abstract

New ideas are often thought to arise from recombining existing knowledge. Yet despite rapid publication growth - and expanding opportunities for recombination - scientific breakthroughs remain rare. This gap between productivity and progress challenges recombinant growth theory as the prevailing account of innovation. We argue that the limitation of this theory lies in treating ideas solely as complements, overlooking that breakthroughs often arise when ideas act as substitutes. To test this, we integrate scientist interviews, bibliometric validation, and machine learning analysis of 41 million papers (1965 - 2024). Interviews reveal that breakthroughs are marked not by novelty (Atypicality) alone but by their ability to displace dominant ideas (Disruption). Large-scale analysis confirms that novelty and disruption represent distinct innovation mechanisms: they are negatively correlated across domains, periods, team sizes, and paper versions. Novel papers extend dominant ideas across topics and attract immediate attention; disruptive papers displace them within the same topic and generate lasting influence. Hence, progress slows not from lack of effort but because most research extends rather than overturns ideas. Applying this perspective reveals distinct roles of theories and methods in scientific change: methods more often drive breakthroughs, whereas theories tend to be novel but rarely disruptive, reinforcing the dominance of established ideas.