Cheap Permutation Testing

TL;DR

This paper introduces a fast permutation testing method that groups data points into bins, significantly reducing computation time while maintaining the statistical properties and power of traditional permutation tests.

Contribution

The authors propose a binning-based permutation test approach that preserves false positive control and power, with theoretical guarantees and practical improvements over existing methods.

Findings

Cheap permutation tests run in time comparable to evaluating a single statistic.

They closely approximate the power of standard permutation tests.

Experiments show improved efficiency over MMD, HSIC, and other tests.

Abstract

Permutation tests are a popular choice for distinguishing distributions and testing independence, due to their exact, finite-sample control of false positives and their minimax optimality when paired with U-statistics. However, standard permutation tests are also expensive, requiring a test statistic to be computed hundreds or thousands of times to detect a separation between distributions. In this work, we offer a simple approach to accelerate testing: group your datapoints into bins and permute only those bins. For U and V-statistics, we prove that these cheap permutation tests have two remarkable properties. First, by storing appropriate sufficient statistics, a cheap test can be run in time comparable to evaluating a single test statistic. Second, cheap permutation power closely approximates standard permutation power. As a result, cheap tests inherit the exact false positive control and minimax optimality of standard permutation tests while running in a fraction of the time. We complement these findings with improved power guarantees for standard permutation testing and experiments demonstrating the benefits of cheap permutations over standard maximum mean discrepancy (MMD), Hilbert-Schmidt independence criterion (HSIC), random Fourier feature, Wilcoxon-Mann-Whitney, cross-MMD, and cross-HSIC tests.