Distributions and Statistical Power of Optimal Signal-Detection Methods In Finite Cases

TL;DR

This paper develops analytical methods to compute p-values and power for a broad family of goodness-of-fit tests, enabling practical application of optimal signal-detection methods in finite-sample scenarios, especially in genomics.

Contribution

It provides a general analytical framework for calculating p-values and power of GOF tests, including HC and B-J, in finite samples, bridging the gap between asymptotic theory and practical data analysis.

Findings

HC performs best for rare signals.

B-J is more robust across different signal patterns.

The R package SetTest facilitates practical application.

Abstract

In big data analysis for detecting rare and weak signals among $n$ features, some grouping-test methods such as Higher Criticism test (HC), Berk-Jones test (B-J), and $\phi$-divergence test share the similar asymptotical optimality when $n \rightarrow \infty$. However, in practical data analysis $n$ is frequently small and moderately large at most. In order to properly apply these optimal tests and wisely choose them for practical studies, it is important to know how to get the p-values and statistical power of them. To address this problem in an even broader context, this paper provides analytical solutions for a general family of goodness-of-fit (GOF) tests, which covers these optimal tests. For any given i.i.d. and continuous distributions of the input test statistics of the $n$ features, both p-value and statistical power of such a GOF test can be calculated. By calculation we compared the finite-sample performances of asymptotically optimal tests under the normal mixture alternative. Results show that HC is the best choice when signals are rare, while B-J is more robust over various signal patterns. In the application to a real genome-wide association study, results illustrate that the p-value calculation works well, and the optimal tests have potentials for detecting novel disease genes with weak genetic effects. The calculations have been implemented in an R package SetTest and published on the CRAN.