Global p-Values in Multi-Design Studies

TL;DR

This paper introduces a unified framework using a global p-value to effectively combine results from multiple analysis strategies in multi-design studies, addressing replicability issues and controlling error rates.

Contribution

It proposes a novel g-value based framework that aggregates outcomes in multi-design studies, improving replicability and statistical validity.

Findings

Mitigates selective reporting risks

Controls type I error rates effectively

Maintains statistical power in analyses

Abstract

Replicability issues -- referring to the difficulty or failure of independent researchers to corroborate the results of published studies -- have hindered the meaningful progression of science and eroded public trust in scientific findings. In response to the replicability crisis, one approach is the use of multi-design studies, which incorporate multiple analysis strategies to address a single research question. However, there remains a lack of methods for effectively combining outcomes in multi-design studies. In this paper, we propose a unified framework based on the g-value, for global p-value, which enables meaningful aggregation of outcomes from all the considered analysis strategies in multi-design studies. Our framework mitigates the risk of selective reporting while rigorously controlling type I error rates. At the same time, it maintains statistical power and reduces the likelihood of overlooking true positive effects. Importantly, our method is flexible and broadly applicable across various scientific domains and outcome results.