More Permutations Do Not Always Increase Power: Non-monotonicity in Monte Carlo Permutation Tests

TL;DR

This paper reveals that increasing the number of permutations in Monte Carlo permutation tests can actually decrease statistical power due to distributional discreteness, challenging common assumptions.

Contribution

It provides a structural explanation and proof that power can non-monotonically decrease as the Monte Carlo sample size increases.

Findings

Power can decrease infinitely often as the number of permutations increases.

Distributional discreteness causes non-monotonicity in test power.

Common intuition that more permutations always improve power is false.

Abstract

Monte Carlo permutation tests are a cornerstone of valid, model-free statistical inference. A widely held practical intuition is that increasing the number of sampled permutations improves test performance, in particular that statistical power tends to increase with the Monte Carlo budget. In this paper, we show that these intuitions are false in general. Leveraging the saw-toothed structure of power arising from distributional discreteness, we provide a simple structural explanation for why power can decrease as the number of sampled permutations increases, and we prove that such decreases occur infinitely often as the Monte Carlo budget grows.