Nonparametric estimation of the density of the alternative hypothesis in a multiple testing setup. Application to local false discovery rate estimation

TL;DR

This paper develops and analyzes two nonparametric estimators for the alternative hypothesis density in a multiple testing framework, improving local false discovery rate estimation with proven asymptotic properties.

Contribution

It introduces the first convergence results for nonparametric estimation of the alternative density in mixture models, using kernel and maximum smoothed likelihood methods.

Findings

Kernel estimator achieves classical nonparametric convergence rate.

Maximum smoothed likelihood estimator has a descent iterative algorithm.

Estimators improve local false discovery rate estimation in simulations.

Abstract

In a multiple testing context, we consider a semiparametric mixture model with two components where one component is known and corresponds to the distribution of $p$-values under the null hypothesis and the other component $f$ is nonparametric and stands for the distribution under the alternative hypothesis. Motivated by the issue of local false discovery rate estimation, we focus here on the estimation of the nonparametric unknown component $f$ in the mixture, relying on a preliminary estimator of the unknown proportion $\theta$ of true null hypotheses. We propose and study the asymptotic properties of two different estimators for this unknown component. The first estimator is a randomly weighted kernel estimator. We establish an upper bound for its pointwise quadratic risk, exhibiting the classical nonparametric rate of convergence over a class of H\"older densities. To our knowledge, this is the first result establishing convergence as well as corresponding rate for the estimation of the unknown component in this nonparametric mixture. The second estimator is a maximum smoothed likelihood estimator. It is computed through an iterative algorithm, for which we establish a descent property. In addition, these estimators are used in a multiple testing procedure in order to estimate the local false discovery rate. Their respective performances are then compared on synthetic data.