Structural Diversity Drives Disruptive Scientific Innovation

TL;DR

This paper introduces Structural Diversity (SD) as a key metric of collaboration architecture that predicts disruptive scientific innovation and enhances team creativity by bridging multiple knowledge communities.

Contribution

It defines SD, demonstrates its predictive power over traditional metrics, and shows how it can be used to design more innovative scientific teams.

Findings

SD outperforms traditional metrics in predicting disruptive innovation

Higher SD interacts positively with team size to foster creativity

Teams with higher SD more effectively combine heterogeneous knowledge

Abstract

Scientific innovation increasingly depends on collaboration, yet the organizational structure that fosters breakthrough ideas remains poorly understood. Existing metrics - such as team size or compositional diversity - capture readily observable characteristics but not the deeper architecture of collaboration. We introduce Structural Diversity (SD): the extent to which a team bridges multiple distinct knowledge communities within its prior collaboration network. Using a century-scale dataset of 260 million scientific publications (1900-2025) and combining causal inference with a quasi-natural experiment based on a U.S. National Science Foundation policy change in 2012, we show that SD is a powerful and robust predictor of disruptive innovation, outperforming traditional team novelty indicators such as team freshness and edge density. Moreover, SD positively interacts with team size and is able to mitigate the well-known "curse of scale" by transforming scale from a liability into a resource for creative synthesis. We find that one mechanism underlying this effect is Disciplinary Integration (DI): teams with higher SD can more effectively combine heterogeneous knowledge into novel configurations. Our findings position SD as both a new theoretical construct and an actionable design principle for organizing scientific collaboration. By linking the architecture of team assembly to the dynamics of creative discovery, our work offers a structural explanation for how collective intelligence can be systematically engineered to foster disruptive innovation.