Idea engines: Unifying innovation and obsolescence from markets and genetic evolution to science

TL;DR

This paper presents a unified model of innovation and obsolescence across biological, technological, and scientific systems, predicting different regimes of system dynamics and aligning with empirical data.

Contribution

It introduces a minimal, first-principles model connecting innovation and obsolescence, revealing regimes and structural patterns in the evolution of the 'space of the possible.'

Findings

Identifies three regimes: finite, growing, and collapsing systems.

Predicts a critical boundary with fluctuating diversity in the system.

Aligns model predictions with empirical data on firms, COVID, and scientific citations.

Abstract

Innovation and obsolescence describe dynamics of ever-churning and adapting social and biological systems, concepts that encompass field-specific formulations. We formalize the connection with a reduced model of the dynamics of the "space of the possible" (e.g. technologies, mutations, theories) to which agents (e.g. firms, organisms, scientists) couple as they grow, die, and replicate. We predict three regimes: the space is finite, ever growing, or a Schumpeterian dystopia in which obsolescence drives the system to collapse. We reveal a critical boundary at which the space of the possible fluctuates dramatically in size, displaying recurrent periods of minimal and of veritable diversity. When the space is finite, corresponding to physically realizable systems, we find surprising structure. This structure predicts a taxonomy for the density of agents near and away from the innovative frontier that we compare with distributions of firm productivity, covid diversity, and citation rates for scientific publications. Remarkably, our minimal model derived from first principles aligns with empirical examples, implying a follow-the-leader dynamic in firm cost efficiency and biological evolution, whereas scientific progress reflects consensus that waits on old ideas to go obsolete. Our theory introduces a fresh and empirically testable framework for unifying innovation and obsolescence across fields.