The Sequential Nature of Science: Quantifying Learning from a Sequence of Studies

TL;DR

This paper introduces SMART, a new method for analyzing the influence of individual studies over time, capturing how scientific knowledge evolves sequentially and how new research can challenge prior beliefs.

Contribution

The paper presents SMART, a novel sequential meta-analysis approach that accounts for the temporal influence of studies, highlighting the impact of methodological innovations often missed by classical methods.

Findings

SMART reveals the influence of studies at the time they are published.

Methodological critiques and innovations significantly affect scientific progress.

Sequential analysis uncovers insights overlooked by traditional meta-analysis.

Abstract

Scientific progress is inherently sequential: collective knowledge is updated as new studies enter the literature. We propose the sequential meta-analysis research trace (SMART), which quantifies the influence of each study at the time it enters the literature. In contrast to classical meta-analysis, our method can capture how new studies may cast doubt on previously held beliefs, increasing collective uncertainty. For example, a new study may present a methodological critique of prior work and propose a superior method. Even small studies, which may not materially affect a retrospective meta-analysis, can be influential at the time they appeared. To contrast SMART with classical meta-analysis, we re-analyze two meta-analysis datasets, from psychology and labor economics. One assembles studies using a single methodology; the other contains studies that predate or follow an important methodological innovation. Our formalization of sequential learning highlights the importance of methodological innovation that might otherwise be overlooked by classical meta-analysis.