Covariance Correction for Permutation Statistics in Multiple Testing Problems

TL;DR

This paper introduces a computationally efficient covariance correction method for permutation tests in multiple testing, ensuring valid joint dependence structure and improving inference accuracy.

Contribution

It proposes a novel, flexible covariance correction technique that overcomes limitations of existing methods like prepivoting, applicable to various permutation testing scenarios.

Findings

The new method accurately restores joint dependence structure in permutation tests.

Simulation studies show the approach outperforms existing methods in diverse settings.

The method is computationally efficient and broadly applicable.

Abstract

In qualitative statistics, permutation tests are very popular, mainly because of their finite-sample exactness under exchangeability. However, in non-exchangeable settings, the covariance structure of permuted statistics typically differs from that of the original statistic. A common solution is studentization, which restores asymptotic correctness for general hypotheses while preserving exactness under exchangeability. In multiple testing settings, however, standard studentization fails to provide the correct joint limiting distribution. Existing solutions such as prepivoting address this issue but are computationally expensive and therefore rarely used in practice. We propose a general, computationally more efficient methodology that overcomes this fundamental limitation. By appropriately correcting the covariance matrix of multiple permutation statistics, our approach restores the correct joint asymptotic dependence structure, enabling asymptotically valid permutation tests in broad multiple testing frameworks. The proposed method is highly flexible: it accommodates singular covariance structures and is not tied to specific parameters, test statistics, or permutation schemes. This generality makes it applicable across a wide range of problems. Extensive simulation studies demonstrate that our approach results in reliable inference and outperforms existing methods across diverse settings.