False discovery rate control with compound p-values

TL;DR

This paper investigates the properties of the Benjamini--Hochberg procedure when applied to compound p-values, revealing bounds on FDR control under independence and dependence, with practical implications for multiple testing scenarios.

Contribution

It provides theoretical bounds on FDR when using compound p-values with the BH procedure, extending understanding of multiple testing under various dependence structures.

Findings

FDR is at most 1.93 times the nominal level under independence.

FDR can reach 7/6 times the nominal level in worst-case distributions.

Under positive dependence, FDR may be inflated by a factor of O(log m).

Abstract

In the setting of multiple testing, compound p-values generalize p-values by asking for superuniformity to hold only \emph{on average} across all true nulls. We study the properties of the Benjamini--Hochberg procedure applied to compound p-values. Under independence, we show that the false discovery rate (FDR) is at most $1.93\alpha$, where $\alpha$ is the nominal level, and exhibit a distribution for which the FDR is $\frac{7}{6}\alpha$. If additionally all nulls are true, then the upper bound can be improved to $\alpha + 2\alpha^2$, with a corresponding worst-case lower bound of $\alpha + \alpha^2/4$. Under positive dependence, on the other hand, we demonstrate that FDR can be inflated by a factor of $O(\log m)$, where~$m$ is the number of hypotheses. We provide numerous examples of settings where compound p-values arise in practice, either because we lack sufficient information to compute non-trivial p-values, or to facilitate a more powerful analysis.