Higher criticism: $p$-values and criticism

TL;DR

This paper compares higher criticism and Berk-Jones statistics for detecting sparse signals, introduces new significance level approximations, and evaluates their power and applications through simulations.

Contribution

It provides new approximations for significance levels and a comparative analysis of higher criticism and Berk-Jones statistics in various applications.

Findings

Berk-Jones statistics outperform higher criticism in power for sparse detection.

New significance level approximations improve accuracy of statistical tests.

Berk-Jones statistics are effective in applications like false hypothesis bounds and copy number variant detection.

Abstract

This paper compares the higher criticism statistic (Donoho and Jin [Ann. Statist. 32 (2004) 962-994]), a modification of the higher criticism statistic also suggested by Donoho and Jin, and two statistics of the Berk-Jones [Z. Wahrsch. Verw. Gebiete 47 (1979) 47-59] type. New approximations to the significance levels of the statistics are derived, and their accuracy is studied by simulations. By numerical examples it is shown that over a broad range of sample sizes the Berk-Jones statistics have a better power function than the higher criticism statistics to detect sparse mixtures. The applications suggested by Meinshausen and Rice [Ann. Statist. 34 (2006) 373-393], to find lower confidence bounds for the number of false hypotheses, and by Jeng, Cai and Li [Biometrika 100 (2013) 157-172], to detect copy number variants, are also studied.