Fast Approximation of Small p-values in Permutation Tests by Partitioning the Permutations

TL;DR

This paper introduces a fast, efficient method for approximating very small permutation p-values in two-sample tests, significantly reducing computational time while maintaining accuracy, especially for extremely small p-values.

Contribution

The authors develop an asymptotic approximation and a resampling algorithm based on permutation partitions, improving computational efficiency for small p-value estimation in permutation tests.

Findings

Resampling algorithm outperforms existing methods for p-values <10^{-30}

Methods successfully identify significant genes in cancer genomic data

Approaches are applicable to difference and ratio of means in two-sample tests

Abstract

Researchers in genetics and other life sciences commonly use permutation tests to evaluate differences between groups. Permutation tests have desirable properties, including exactness if data are exchangeable, and are applicable even when the distribution of the test statistic is analytically intractable. However, permutation tests can be computationally intensive. We propose both an asymptotic approximation and a resampling algorithm for quickly estimating small permutation p-values (e.g. $<10^{-6}$) for the difference and ratio of means in two-sample tests. Our methods are based on the distribution of test statistics within and across partitions of the permutations, which we define. In this article, we present our methods and demonstrate their use through simulations and an application to cancer genomic data. Through simulations, we find that our resampling algorithm is more computationally efficient than another leading alternative, particularly for extremely small p-values (e.g. $<10^{-30}$). Through application to cancer genomic data, we find that our methods can successfully identify up- and down-regulated genes. While we focus on the difference and ratio of means, we speculate that our approaches may work in other settings.