Optimal testing using combined test statistics across independent studies

TL;DR

This paper develops a theoretical framework for meta-analysis test power in high-dimensional models, quantifies the efficiency loss of common combination methods, and proposes strategies for optimal test aggregation across independent studies.

Contribution

It introduces a mathematical framework for meta-analysis in high-dimensional models, deriving bounds and optimal strategies for combining test statistics.

Findings

Quantifies the loss of standard meta-analysis methods compared to pooled data.

Identifies an elbow effect where local optimal tests can be sub-optimal for meta-analysis.

Proposes approaches to achieve global optimality and explores benefits of limited trial coordination.

Abstract

Combining test statistics from independent trials or experiments is a popular method of meta-analysis. However, there is very limited theoretical understanding of the power of the combined test, especially in high-dimensional models considering composite hypotheses tests. We derive a mathematical framework to study standard {meta-analysis} testing approaches in the context of the many normal means model, which serves as the platform to investigate more complex models. We introduce a natural and mild restriction on the meta-level combination functions of the local trials. This allows us to mathematically quantify the cost of compressing $m$ trials into real-valued test statistics and combining these. We then derive minimax lower and matching upper bounds for the separation rates of standard combination methods for e.g. p-values and e-values, quantifying the loss relative to using the full, pooled data. We observe an elbow effect, revealing that in certain cases combining the locally optimal tests in each trial results in a sub-optimal {meta-analysis} method and develop approaches to achieve the global optima. We also explore the possible gains of allowing limited coordination between the trial designs. Our results connect meta-analysis with bandwidth constraint distributed inference and build on recent information theoretic developments in the latter field.