Quantifying How Much Has Been Learned from a Research Study

TL;DR

This paper introduces a Bayesian learning metric using Wasserstein-2 distance to quantify how much a research study advances scientific knowledge by updating community beliefs, offering a formal and transparent evaluation tool.

Contribution

It proposes a novel Bayesian framework and Wasserstein-2 distance-based metric to measure and compare the knowledge gained from research studies, including prospective assessments.

Findings

Demonstrates the metric through stylized examples

Shows how the approach differs from traditional significance testing

Provides empirical applications illustrating the method's utility

Abstract

How much does a research study contribute to a scientific literature? We propose a learning metric to quantify how much a research community learns from a given study. To do so, we adopt a Bayesian perspective and assess changes in the community's beliefs once updated with a new study's evidence. We recommend the Wasserstein-2 distance as a way to describe how the research community's prior beliefs change to incorporate a study's findings. We illustrate this approach through stylized examples and empirical applications, showing how it differs from more traditional evaluative standards, such as statistical significance. We then extend the framework to the prospective setting, offering a way for decision-makers to evaluate the expected amount of learning from a proposed study. While assessments about what has or could be learned from a research program are often expressed informally, our learning metric provides a principled tool for judging scientific contributions. By formalizing these judgments, our measure has the potential to allow for more transparent assessments of past and prospective research contributions.