Meta-Analysis with Untrusted Data

TL;DR

This paper introduces a novel meta-analysis method that incorporates untrusted observational data and heterogeneous trials using conformal prediction, resulting in more precise and reliable causal effect estimates in healthcare.

Contribution

It develops a conformal prediction-based framework that handles untrusted data and heterogeneity in meta-analysis without strong assumptions.

Findings

Tighter, more reliable prediction intervals in healthcare datasets.

Effective incorporation of untrusted observational data.

Handling heterogeneity improves meta-analytic precision.

Abstract

[See paper for full abstract] Meta-analysis is a crucial tool for answering scientific questions. It is usually conducted on a relatively small amount of ``trusted'' data -- ideally from randomized, controlled trials -- which allow causal effects to be reliably estimated with minimal assumptions. We show how to answer causal questions much more precisely by making two changes. First, we incorporate untrusted data drawn from large observational databases, related scientific literature and practical experience -- without sacrificing rigor or introducing strong assumptions. Second, we train richer models capable of handling heterogeneous trials, addressing a long-standing challenge in meta-analysis. Our approach is based on conformal prediction, which fundamentally produces rigorous prediction intervals, but doesn't handle indirect observations: in meta-analysis, we observe only noisy effects due to the limited number of participants in each trial. To handle noise, we develop a simple, efficient version of fully-conformal kernel ridge regression, based on a novel condition called idiocentricity. We introduce noise-correcting terms in the residuals and analyze their interaction with a ``variance shaving'' technique. In multiple experiments on healthcare datasets, our algorithms deliver tighter, sounder intervals than traditional ones. This paper charts a new course for meta-analysis and evidence-based medicine, where heterogeneity and untrusted data are embraced for more nuanced and precise predictions.