Bayesian Nonparametric Sensitivity Analysis of Multiple Test Procedures Under Dependence

TL;DR

This paper develops a Bayesian nonparametric sensitivity analysis method for multiple testing procedures using Dirichlet process priors, allowing for uncertainty quantification under arbitrary dependence among p-values.

Contribution

It introduces a novel DP-based sensitivity analysis framework for MTPs that accounts for dependence and reduces conservativeness in multiple testing.

Findings

Applied to a large U.S. high school dataset with 28,000 p-values.

Demonstrated uncertainty quantification in multiple testing decisions.

Provided software implementation in R.

Abstract

This article introduces a sensitivity analysis method for Multiple Testing Procedures (MTPs) using marginal $p$-values. The method is based on the Dirichlet process (DP) prior distribution, specified to support the entire space of MTPs, where each MTP controls either the family-wise error rate (FWER) or the false discovery rate (FDR) under arbitrary dependence between $p$-values. This DP MTP sensitivity analysis method provides uncertainty quantification for MTPs, by accounting for uncertainty in the selection of such MTPs and their respective threshold decisions regarding which number of smallest $p$-values are significant discoveries, from a given set of null hypothesis tested, while measuring each $p$-value's probability of significance over the DP prior predictive distribution of this space of all MTPs, and reducing the possible conservativeness of using only one such MTP for multiple testing. The DP MTP sensitivity analysis method is illustrated through the analysis of over twenty-eight thousand $p$-values arising from hypothesis tests performed on a 2022 dataset of a representative sample of three million U.S. high school students observed on 239 variables. They include tests which, respectively, relate variables about the disruption caused by school closures during the COVID-19 pandemic, with various mathematical cognition, academic achievement, and student background variables. R software code for the DP MTP sensitivity analysis method is provided in the Code and Data Supplement of this article (available upon request).