Nonparametric methods controlling the median of the false discovery proportion

TL;DR

This paper introduces a nonparametric multiple testing method that effectively controls the median false discovery proportion, especially useful when most hypotheses are expected to be false, with broad applicability beyond safety testing.

Contribution

It proposes a novel nonparametric approach for controlling the median FDP that is valid under dependence and applicable to various parameters, improving power in specific testing scenarios.

Findings

Method controls the median of the FDP in finite samples.

Approach is valid under dependence among test statistics.

Often has superior power with one-sided tests.

Abstract

When testing many hypotheses, often we do not have strong expectations about the directions of the effects. In some situations however, the alternative hypotheses are that the parameters lie in a certain direction or interval, and it is in fact expected that most hypotheses are false. This is often the case when researchers perform multiple noninferiority or equivalence tests, e.g. when testing food safety with metabolite data. The goal is then to use data to corroborate the expectation that most hypotheses are false. We propose a nonparametric multiple testing approach that is powerful in such situations. If the user's expectations are wrong, our approach will still be valid but have low power. Of course all multiple testing methods become more powerful when appropriate one-sided instead of two-sided tests are used, but our approach often has superior power then. The proposed methods are not at all limited to safety testing and can be used for testing hypotheses about various kinds of parameters, such as coefficients of a model. The methods in this paper control the median of the false discovery proportion (FDP), which is the fraction of false discoveries among the rejected hypotheses. This approach is comparable to false discovery rate control, where one ensures that the mean rather than the median of the FDP is small. Our procedures make use of a symmetry property of the test statistics, do not require independence and have finite-sample properties.