Higher Criticism to Compare Two Large Frequency Tables, with sensitivity to Possible Rare and Weak Differences

TL;DR

This paper adapts the Higher Criticism test to compare large frequency tables, effectively detecting rare and weak differences by analyzing the asymptotic power and phase transition behavior in various data regimes.

Contribution

It introduces a novel HC-based method for comparing large frequency tables, characterizing its asymptotic power and phase transition in sparse and dense data settings.

Findings

HC test achieves maximal power in a specific region of the parameter space.

The phase transition curve in high-counts matches that of the two-sample normal means model.

The method effectively detects rare and weak differences in large frequency tables.

Abstract

We adapt Higher Criticism (HC) to the comparison of two frequency tables which may -- or may not -- exhibit moderate differences between the tables in some unknown, relatively small subset out of a large number of categories. Our analysis of the power of the proposed HC test quantifies the rarity and size of assumed differences and applies moderate deviations-analysis to determine the asymptotic powerfulness/powerlessness of our proposed HC procedure. Our analysis considers the null hypothesis of no difference in underlying generative model against a rare/weak perturbation alternative, in which the frequencies of $N^{1-\beta}$ out of the $N$ categories are perturbed by $r(\log N)/2n$ in the Hellinger distance; here $n$ is the size of each sample. Our proposed Higher Criticism (HC) test for this setting uses P-values obtained from $N$ exact binomial tests. We characterize the asymptotic performance of the HC-based test in terms of the rarity parameter $\beta$ and the perturbation intensity parameter $r$. Specifically, we derive a region in the $(\beta,r)$-plane where the test asymptotically has maximal power, while having asymptotically no power outside this region. Our analysis distinguishes between cases in which the counts in both tables are low, versus cases in which counts are high, corresponding to the cases of sparse and dense frequency tables. The phase transition curve of HC in the high-counts regime matches formally the curve delivered by HC in a two-sample normal means model.