An algorithm to compute the power of Monte Carlo tests with guaranteed precision

TL;DR

This paper introduces a novel algorithm that provides guaranteed, conservative confidence intervals for the power of Monte Carlo tests, adaptable to various test types and ensuring specified precision and coverage.

Contribution

It presents the first method capable of computing guaranteed confidence intervals for Monte Carlo test power with adaptable effort and broad applicability.

Findings

Algorithm achieves finite expected effort in practical scenarios.

Provides conservative confidence intervals with user-specified length and coverage.

Implemented in R package simctest, available on CRAN.

Abstract

This article presents an algorithm that generates a conservative confidence interval of a specified length and coverage probability for the power of a Monte Carlo test (such as a bootstrap or permutation test). It is the first method that achieves this aim for almost any Monte Carlo test. Previous research has focused on obtaining as accurate a result as possible for a fixed computational effort, without providing a guaranteed precision in the above sense. The algorithm we propose does not have a fixed effort and runs until a confidence interval with a user-specified length and coverage probability can be constructed. We show that the expected effort required by the algorithm is finite in most cases of practical interest, including situations where the distribution of the p-value is absolutely continuous or discrete with finite support. The algorithm is implemented in the R-package simctest, available on CRAN.