Towards more scientific meta-analyses

TL;DR

This paper advocates for response-surface meta-analysis, a method that estimates the true scientific effect by modeling study quality and effect size, addressing limitations of traditional meta-analyses.

Contribution

It reintroduces and provides a practical implementation of response-surface meta-analysis, focusing on estimating the true scientific effect rather than the average effect of imperfect studies.

Findings

Response-surface meta-analysis aligns more closely with scientific effects.

Compared to traditional methods, it offers more accurate effect size estimates.

Application to real-world data demonstrates its practical utility.

Abstract

Meta-analysis can be a critical part of the research process, often serving as the primary analysis on which the practitioners, policymakers, and individuals base their decisions. However, current literature synthesis approaches to meta-analysis typically estimate a different quantity than what is implicitly intended; concretely, standard approaches estimate the average effect of a treatment for a population of imperfect studies, rather than the true scientific effect that would be measured in a population of hypothetical perfect studies. We advocate for an alternative method, called response-surface meta-analysis, which models the relationship between the quality of the study design as predictor variables and its reported estimated effect size as the outcome variable in order to estimate the effect size obtained by the hypothetical ideal study. The idea was first introduced by Rubin several decades ago, and here we provide a practical implementation. First, we reintroduce the idea of response-surface meta-analysis, highlighting its focus on a scientifically-motivated estimand while proposing a straightforward implementation. Then we compare the approach to traditional meta-analysis techniques used in practice. We then implement response-surface meta-analysis and contrast its results with existing literature-synthesis approaches on both simulated data and a real-world example published by the Cochrane Collaboration. We conclude by detailing the primary challenges in the implementation of response-surface meta-analysis and offer some suggestions to tackle these challenges.