A Genetic Algorithm-based Framework for Learning Statistical Power Manifold

TL;DR

This paper introduces a genetic algorithm-based framework that efficiently learns the statistical power manifold for hypothesis testing, significantly reducing computational costs compared to brute-force methods.

Contribution

It presents a novel genetic algorithm approach to rapidly learn statistical power manifolds, improving efficiency over traditional simulation-based methods.

Findings

Faster learning of power manifolds with fewer queries

Improved manifold quality with more iterations

Effective for multiple linear regression F-tests

Abstract

Statistical power is a measure of the replicability of a categorical hypothesis test. Formally, it is the probability of detecting an effect, if there is a true effect present in the population. Hence, optimizing statistical power as a function of some parameters of a hypothesis test is desirable. However, for most hypothesis tests, the explicit functional form of statistical power for individual model parameters is unknown; but calculating power for a given set of values of those parameters is possible using simulated experiments. These simulated experiments are usually computationally expensive. Hence, developing the entire statistical power manifold using simulations can be very time-consuming. We propose a novel genetic algorithm-based framework for learning statistical power manifolds. For a multiple linear regression $F$-test, we show that the proposed algorithm/framework learns the statistical power manifold much faster as compared to a brute-force approach as the number of queries to the power oracle is significantly reduced. We also show that the quality of learning the manifold improves as the number of iterations increases for the genetic algorithm. Such tools are useful for evaluating statistical power trade-offs when researchers have little information regarding a priori best guesses of primary effect sizes of interest or how sampling variability in non-primary effects impacts power for primary ones.