Data volume and power of multiple tests with small sample size per null

TL;DR

This paper investigates how data volume and power behave in multiple hypothesis testing with small sample sizes per null, revealing that data volume must grow rapidly and power diminishes as null differences become subtle.

Contribution

It provides theoretical insights into the relationship between data volume, power, and false null detection in small-sample multiple testing scenarios.

Findings

Data volume must grow faster than in large-sample cases to detect subtle effects.

Power must decay to zero to control pFDR asymptotically in small-sample settings.

No asymptotically optimal procedure exists for controlling pFDR at the same level.

Abstract

In multiple hypothesis testing, the volume of data, defined as the number of replications per null times the total number of nulls, usually defines the amount of resource required. On the other hand, power is an important measure of performance for multiple testing. Due to practical constraints, the number of replications per null may not be large enough in terms of the difference between false and true nulls. For the case where the population fraction of false nulls is constant, we show that, as the difference between false and true nulls becomes increasingly subtle while the number of replications per null cannot increase fast enough, (1) in order to have enough chance that the data to be collected will yield some trustworthy findings, as measured by a conditional version of the positive false discovery rate (pFDR), the volume of data has to grow at a rate much faster than in the case where the number of replications per null can be large enough, and (2) in order to control the pFDR asymptotically, power has to decay to 0 in a rate highly sensitive to rejection criterion and there is no asymptotically most powerful procedures among those that control the pFDR asymptotically at the same level.