Single-Dataset Meta-Analysis For Many-Analysts And Multiverse Studies

TL;DR

This paper introduces a novel single-dataset meta-analysis method that synthesizes results from multiple analyses on the same dataset, preventing overconfidence and enabling robust inference across diverse analytic choices.

Contribution

It presents a weighted-likelihood approach for meta-analyzing multiple analyses from the same dataset, including classical and Bayesian inference, with practical software implementation.

Findings

Provides meta-analytic estimates of average effects across analytic choices.

Quantifies between-analyst heterogeneity in results.

Demonstrates applicability through real-world case studies.

Abstract

Empirical claims often rely on one population, design, and analysis. Many-analysts, multiverse, and robustness studies expose how results can vary across plausible analytic choices. Synthesizing these results, however, is nontrivial as all results are computed from the same dataset. We introduce single-dataset meta-analysis, a weighted-likelihood approach that incorporates the information in the dataset at most once. It prevents overconfident inferences that would arise if a standard meta-analysis was applied to the data. Single-dataset meta-analysis yields meta-analytic point and interval estimates of the average effect across analytic approaches and of between-analyst heterogeneity, and can be supplied by classical and Bayesian hypothesis tests. Both the common-effect and random-effects versions of the model can be estimated by standard meta-analytic software with small input adjustments. We demonstrate the method via application to the many-analysts study on racial bias in soccer, the many-analysts study of marital status and cardiovascular disease, and the multiverse study on technology use and well-being. The results show how single-dataset meta-analysis complements the qualitative evaluation of many-analysts and multiverse studies.