Adaptive estimation of the sparsity in the Gaussian vector model

TL;DR

This paper develops adaptive methods for estimating and testing the sparsity level in Gaussian vector models, providing minimax optimal procedures and extending to unknown variance scenarios.

Contribution

It introduces a new adaptive testing procedure for sparsity estimation and a novel framework for assessing estimator optimality in Gaussian models.

Findings

Established minimax separation distances for sparsity testing.

Proposed a minimax adaptive test for sparsity.

Provided a new approach to evaluate the optimality of sparsity estimators.

Abstract

Consider the Gaussian vector model with mean value {\theta}. We study the twin problems of estimating the number |{\theta}|_0 of non-zero components of {\theta} and testing whether |{\theta}|_0 is smaller than some value. For testing, we establish the minimax separation distances for this model and introduce a minimax adaptive test. Extensions to the case of unknown variance are also discussed. Rewriting the estimation of |{\theta}|_0 as a multiple testing problem of all hypotheses {|{\theta}|_0 <= q}, we both derive a new way of assessing the optimality of a sparsity estimator and we exhibit such an optimal procedure. This general approach provides a roadmap for estimating the complexity of the signal in various statistical models.

Tables

Table 1. Table 1: Square minimax separation distances (in the ≍ γ subscript asymptotically-equals 𝛾 \asymp_{\gamma} sense) when the noise level σ 𝜎 \sigma is known for all k 0 ∈ [ 0 , n − 1 ] subscript 𝑘 0 0 𝑛 1 k_{0}\in[0,n-1] and Δ ∈ [ 1 , n − k 0 ] Δ 1 𝑛 subscript 𝑘 0 \Delta\in[1,n-k_{0}] .

$k_0$	$\Delta$	${\rho }_{\gamma }^{\ast 2}[k_0,\Delta ]/{\sigma }^2$
$k_0\le \sqrt{n}$	$1\le \Delta \le n-k_0$	$\Delta \log \left(1+\frac{\sqrt{n}}{\Delta }\right)$
	$\sqrt{n}\le \Delta \le n-k_0$	$\sqrt{n}$
$k_0>\sqrt{n}$	$1\le \Delta \le \sqrt{n^{1/2}{k}_0}$	$\Delta \log \left(1+\frac{k_0}{\Delta }\right)$
	$\sqrt{n^{1/2}{k}_0}\le \Delta \le k_0$	$\Delta \frac{{\log }^2\left(1+\frac{k_0}{\Delta }\right)}{\log \left(1+\frac{k_0}{\sqrt{n}}\right)}$
	$k_0\le \Delta \le n-k_0$	$\frac{k_0}{\log \left(1+\frac{k_0}{\sqrt{n}}\right)}$

Table 2. Table 2: Square minimax separation distance ρ γ , var ∗ 2 ​ [ k 0 , Δ ] superscript subscript 𝜌 𝛾 var absent 2 subscript 𝑘 0 Δ \rho_{\gamma,\mathrm{var}}^{*2}[k_{0},\Delta] (as defined in Equation ( 41 )) when the noise level σ 𝜎 \sigma is unknown but belongs to some known fixed interval [ σ − , σ + ] subscript 𝜎 subscript 𝜎 [\sigma_{-},\sigma_{+}] . Here, c ∈ ( 0 , 1 ) 𝑐 0 1 c\in(0,1) is some fixed universal constant and ξ ∈ ( 0 , 1 ) 𝜉 0 1 \xi\in(0,1) can be chosen arbitrarily small.

$k_0$	$\Delta$	${\rho }_{\gamma ,\mathrm{var}}^{\ast 2}[k_0,\Delta ]/{\sigma }_{+}^2$
$0\le k_0\le \sqrt{n}$	$0\le \Delta \le \sqrt{n}$	$\Delta \log \left(1+\frac{\sqrt{n}}{\Delta }\right)$
		$\sqrt{\Delta n^{1/2}}$
$n^{1-\xi }\ge k_0\ge \sqrt{n}$	$0\le \Delta \le \sqrt{k_0{n}^{1/2}}$	$\Delta \log \left(1+\frac{k_0}{\Delta }\right)$
		$\Delta \frac{{\log }^2\left(1+\frac{k_0}{\Delta }\right)}{\log \left(1+\frac{k_0}{\sqrt{n}}\right)}$
		$\frac{\sqrt{\Delta k_0}}{\log \left(1+\frac{k_0}{\sqrt{n}}\right)}$

Equations

$$Y_i={\theta }_i+{ϵ}_i,i=1,\dots ,n,$$

$${\mathbb{B}}_0[k_0]:=\{\theta \in {\mathbb{R}}^n:\Vert \theta {\Vert }_0\le k_0\},$$

$${\mathbb{B}}_0[k_1,k_0,\rho ]:=\{\theta \in {\mathbb{B}}_0[k_1]:d_2(\theta ,{\mathbb{B}}_0[k_0])\ge \rho \},$$

$$H_{k_0}:\theta \in {\mathbb{B}}_0[k_0]\text{ versus }{H}_{\Delta ,k_0,\rho }:\theta \in {\mathbb{B}}_0[k_0+\Delta ,k_0,\rho ].$$

$$R(T;k_0,\Delta ,\rho ):=\underset{\theta \in {\mathbb{B}}_0[k_0]}{\sup }{P}_{\theta ,\sigma }[T=1]+\underset{\theta \in {\mathbb{B}}_0[k_0+\Delta ,k_0,\rho ]}{\sup }{P}_{\theta ,\sigma }[T=0].$$

$${\rho }_{\gamma }(T;k_0,\Delta ):=\sup \left\{\rho >0|R(T;k_0,\Delta ,\rho )>\gamma \right\}.$$

$${\rho }_{\gamma }^{\ast }[k_0,\Delta ]:=\underset{T}{\inf }{\rho }_{\gamma }(T;k_0,\Delta ),$$

$$\rho_{\gamma}^{*2}[0,\Delta]\asymp_{\gamma}\sigma^{2}\Delta\log\big{(}1+\frac{\sqrt{n}}{\Delta}\big{)}\ ,$$

$$\rho_{\gamma}^{*2}[k_{0},\Delta]\geq\sigma^{2}\Delta\log\Big{[}1+\frac{\sqrt{n}}{8\Delta}\Big{]}\ .$$

$$\rho_{\gamma}^{*2}[k_{0},\Delta]\geq c\sigma^{2}\left\{\begin{array}[]{cc}\Delta\left[\frac{\log^{2}\big{[}1+\frac{k_{0}}{\Delta}\big{]}}{\log\big{[}1+\frac{k_{0}}{\sqrt{n}}\big{]}}\wedge\log\big{[}1+\frac{k_{0}}{\Delta}\big{]}\right]&\text{ if $\Delta\leq k_{0}\wedge(n-k_{0})$}\\ \frac{k_{0}}{\log\big{[}1+\frac{k_{0}}{\sqrt{n}}\big{]}}&\text{ if }k_{0}<\Delta\leq n-k_{0}\end{array}\right.$$

$$N_t:=\#\{i,|Y_i|\ge t\},$$

$$N_{\sigma t}\ge k_0+2(n-k_0)\Phi (t)+u_{t,\alpha }^{HC},$$

$$u_{t,\alpha }^{HC}:=2\sqrt{n\Phi (t)\log \left(\frac{t^2{\pi }^2}{3\alpha }\right)}+\frac{2}{3}\log \left(\frac{t^2{\pi }^2}{3\alpha }\right).$$

$$|\theta_{(k_{0}+q)}|\geq c_{\alpha,\beta}\sigma\sqrt{\log\Big{(}2+\frac{\sqrt{n}\vee k_{0}}{q}\Big{)}}\ ,\quad\text{ for some }q\in[1,n-k_{0}]$$

$${\kappa }_s(x):={\int }_{-1}^1(1-|\xi |)e^{s^2{\xi }^2/2}\cos (s\xi x)d\xi ,$$

$$Z(s):=\sum_{i=1}^{n}\big{(}1-\kappa_{s}(Y_{i}/\sigma)\big{)}\ .$$

$${\overline{\phi }}_n(s):=n^{-1}\sum _{i=1}^n\cos (s{Y}_i),\overline{\phi }(s):=n^{-1}\sum _{i\le n}\cos (s{\theta }_i)e^{-\frac{s^2{\sigma }^2}{2}},$$

$$E_{\theta }[Z(s)]=\sum _{i=1}^{n}1-{\int }_{-1}^1(1-|\xi |)\cos (s\xi {\theta }_i/\sigma )d\xi =\sum _{i=1}^{n}1-2\frac{1-\cos (s{\theta }_i/\sigma )}{(s{\theta }_i/\sigma {)}^2},$$

$$Z(s_{k_0})\ge k_0+u_{k_0,\alpha }^B,\text{where}{u}_{k_0,\alpha }^B:=\frac{e^{s_{k_0}^2/2}}{s_{k_0}}\sqrt{8n\log (2/\alpha )}.$$

$$|{\theta }_{(k_0+q)}|$$

$$\displaystyle\sum_{i=1}^{n}\Big{[}\theta_{i}^{2}\wedge s^{-2}_{k_{0}}\Big{]}$$

$${\eta }_{r,w}(x):=\frac{r}{(1-2\Phi (r))}{\int }_{-1}^1\frac{e^{-r^2{\xi }^2/2}}{\sqrt{2\pi }}{e}^{{\xi }^2{w}^2/2}\cos (\xi wx)d\xi .$$

$$V(r,w):=\sum _{i=1}^{n}1-{\eta }_{r,w}(Y_i/\sigma ).$$

$$E[{\eta }_{r,w}(X)]=\frac{1}{1-2\Phi (r)}{\int }_{-r}^r\varphi (\xi )\cos (\xi x\frac{w}{r})d\xi ,$$

$$r_{k_0,l}:=\sqrt{2\log (\frac{k_0}{l})},w_l:=\sqrt{\log (\frac{l}{\sqrt{n}})}.$$

$$V(r_{k_{0},l},w_{l})\geq k_{0}+l+u^{I}_{k_{0},l,\alpha}\,\quad\quad\text{ where }\quad u^{I}_{k_{0},l,\alpha}:=\sqrt{2ln^{1/2}\log\Big{(}\frac{\pi^{2}[1+\log_{2}(l/l_{k_{0}})]^{2}}{6\alpha}\Big{)}}\ ,$$

$$|{\theta }_{(k_0+q)}|\ge c_{\alpha ,\beta }^{{}^{\prime }}\sigma \frac{1+\log (1+\frac{k_0}{q})}{\sqrt{\log (1+\frac{k_0}{\sqrt{n}})}},\text{ for some }q\ge c_{\alpha ,\beta }^{{}^{\prime \prime }}\sqrt{k_0{n}^{1/2}},$$

$$\psi^{2}_{k_{0},q}:=\left\{\begin{array}[]{cc}\log\Big{[}1+\frac{\sqrt{n}}{q}\Big{]}&\text{if }k_{0}\leq\sqrt{n}\ ,\\ \frac{\log^{2}\big{(}1+\tfrac{k_{0}}{q}\big{)}}{\log\big{(}1+\frac{k_{0}}{\sqrt{n}}\big{)}}\bigwedge\log\big{(}1+\frac{k_{0}}{q}\big{)}&\text{if }k_{0}>\sqrt{n}\text{ and }q\leq k_{0}\ ,\\ \frac{k_{0}}{q\log\big{(}1+\frac{k_{0}}{\sqrt{n}}\big{)}}&\text{if }k_{0}>\sqrt{n}\text{ and }q>k_{0}\ .\end{array}\right.$$

$$|{\theta }_{(k_0+q)}|\ge c_{\alpha ,\beta }\sigma {\psi }_{k_0,q},\text{ for some }q\in [n-k_0].$$

$$\theta \in {\mathbb{B}}_0(k_0+\Delta )\text{ and }{d}^2[\theta ,{\mathbb{B}}_0(k_0)]\ge c_{\alpha ,\beta }^{\prime }{\sigma }^2\Delta {\psi }_{k_0,\Delta }^2,\text{for some Δ∈[n−k0].}$$

$$\hat{k}:=\lceil {\hat{k}}_{HC}\rceil \vee \lceil {\hat{k}}_B\rceil \vee \lceil {\hat{k}}_I\rceil$$

Full text

1 Introduction

Many estimation methods in high or infinite-dimensional statistics rely on the assumption that the parameter of interest belongs to some smaller parameter space. Depending on the problem at hand, the assumptions on the structure of the unknown parameter take various forms. In high-dimensional linear regression, it is usually assumed that the regression parameter is sparse [6]. In matrix completion, the underlying matrix may be supposed to be low-rank [30]. In density estimation, many nonparametric methods are based on the assumption that the function satisfies some smoothness properties [21]. Many Model-based clustering methods require the data to follow a mixture distribution with several Gaussian components [22]. In practice, the exact complexity of the parameter (e.g.  the rank of the matrix, the smoothness of the function) is unknown. Although a lot of work has been devoted to the construction of statistical procedures performing as well as if the model complexity was known (e.g.  [6, 34, 20]), the literature on the estimation of the complexity of the parameter is scarcer.

In this paper, we are interested in the twin problems of estimating the complexity of the parameter and testing whether the parameter belongs to some complexity class. There are several motivations for these problems. First, complexity estimation allows to assess the relevance of specific parameter estimation approaches. For instance, inferring the smoothness of a function allows to justify the use of regularity-based procedures. Second, the construction of adaptive confidence regions is closely connected to the model testing problem since the size of a good confidence region should depend on the complexity of the unknown parameter [23]. Finally, in some practical applications, the primary objective is rather to evaluate the complexity of the parameter than the parameter itself. This is for instance the case in some heritability studies where the goal is to decipher whether a trait is multigenic or “highly polygenic” which amounts to inferring whether a high-dimensional regression parameter is sparse or dense [33, 40].

In this paper, we focus on a comparatively simple, yet emblematic setting, namely the Gaussian vector model, that we define as follows :

[TABLE]

where $\theta=(\theta_{i})\in\mathbb{R}^{n}$ is unknown and the noise components $\epsilon_{i}$ are independent and follow a centered normal distributions with variance $\sigma^{2}$. We are interested in (i) estimating the number $\|\theta\|_{0}$ of non-zero components of $\theta$ and (ii) given some non-negative integer $k_{0}$, testing whether $\|\theta\|_{0}\leq k_{0}$ or $\|\theta\|_{0}>k_{0}$. The former problem is called sparsity estimation and the latter sparsity testing.

1.1 Sparsity testing and separation distances

As the sparsity testing problem is easier to formalize than the sparsity estimation problem, let us be first more specific about it. Given a non-negative integer $k_{0}\in[0,n]$, we write

[TABLE]

for the set of $k_{0}$-sparse vectors $\theta$, that is to say the set of vectors $\theta$ with less than $k_{0}$ non-zero coefficients. Our goal is to test whether $\theta$ belongs to $\mathbb{B}_{0}[k_{0}]$ or not. In order to assess the performances of a test, we need to specify a rejection region and a risk. Before describing our results and the literature, we shall first define the notion of minimax separation distance of a test.

Let $\|.\|_{2}$ stand for the Euclidean norm in $\mathbb{R}^{n}$. For any $\theta\in\mathbb{R}^{n}$, we write $d_{2}(\theta,\mathbb{B}_{0}[k_{0}]):=\inf_{u\in\mathbb{B}_{0}[k_{0}]}\|\theta-u\|_{2}$ for the distance of $\theta$ to the set of $k_{0}$-sparse vectors. Intuitively, any $\alpha$-level test $T$ of the null hypothesis $\{\theta\in\mathbb{B}_{0}[k_{0}]\}$ cannot reject the null with high probability when the true parameter is arbitrarily close (in the $d_{2}(\theta,\mathbb{B}_{0}[k_{0}])$ sense) to $\mathbb{B}_{0}[k_{0}]$. Conversely, any reasonable test should reject the null hypothesis with high probability for parameters $\theta$ that are really distant to $\mathbb{B}_{0}[k_{0}]$. In order to quantify the performances of a given test $T$, it is then classical [3, 25] to rely on the notion of separation distance. Given positive integers $k_{1}>k_{0}$ and a real number $\rho>0$, define

[TABLE]

as the set of $k_{1}$-sparse vectors that lie at distance larger than $\rho$ from the null. Then, for a fixed $\Delta>0$ and $\rho>0$, we consider the testing problem

[TABLE]

The purpose of this definition is to remove from the alternative hypothesis parameters $\theta$ that are too close to the null hypothesis. Given a test $T$, its risk $R(T;k_{0},\Delta,\rho)$ for the above problem (4) is defined as the sum of the type I and type II error probabilities

[TABLE]

Here, $\operatorname{\mathbb{P}}_{\theta,\sigma}$ stands for the distribution of $Y$. The function $\rho\mapsto R(T;k_{0},\Delta,\rho)$ is non-increasing and equals at least one for $\rho=0$. Fixing some $\gamma\in(0,1)$, the separation distance $\rho_{\gamma}(T;k_{0},\Delta)$ is the largest $\rho$ such that the hypotheses

[TABLE]

The separation distance of a good test $T$ should be the smallest possible. Finally, the minimax separation distance is

[TABLE]

where the infimum is taken over all tests $T$. In other words, $\rho^{*}_{\gamma}[k_{0},\Delta]$ is the minimal distance to $\mathbb{B}_{0}[k_{0}]$ such some test is able to reliably distinguish parameters in $\mathbb{B}_{0}[k_{0}]$ from parameters in $\mathbb{B}_{0}[k_{0}+\Delta,k_{0},\rho]$. Hence, it characterizes the difficulty of the testing problem. A test $T$ whose separation distance $\rho_{\gamma}(T;k_{0},\Delta)$ is (up to a multiplicative constant) smaller than $\rho^{*}_{\gamma}[k_{0},\Delta]$ is said to be minimax.

1.2 Our contribution

Our contribution is threefold:

1. (i)

We first establish the minimax separation distance $\rho_{\gamma}^{*}[k_{0},\Delta]$ for all integers $k_{0}$ and all $\Delta>0$. Besides, we introduce a new test which is minimax adaptive for all $\Delta$. 2. (ii)

In the more realistic setting where the noise level $\sigma$ is unknown, the minimax separation distance $\rho_{\gamma,\mathrm{var}}^{*}[k_{0},\Delta]$ (defined in Section 4) is established and minimax adaptive tests are exhibited. Interestingly, it is proved that the sparsity testing problem under unknown noise level is no more difficult than under known noise level for small $\Delta$. For large $\Delta$, the knowledge of $\sigma$ plays an important role. 3. (iii)

We reformulate the sparsity estimation problem as a multiple testing problem where we simultaneously consider all nested hypotheses $H_{q}$ for $q\in[0,n]$. Introducing a multiple testing procedure which is simultaneously optimal over all $q$, we derive an estimator $\widehat{k}$ which is smaller or equal to $\|\theta\|_{0}$ with high probability and is also closest to $\|\theta\|_{0}$ in a minimax sense. Interestingly, this property will be valid for all possible $\theta\in\mathbb{R}^{n}$ and avoids us to rely on any particular assumption on the parameter. More generally, this perspective also provides a general roadmap to handle the problem of complexity estimation using simultaneous separation distances.

Before discussing more specifically these three points, let us review the literature.

1.3 Related literature

Although the twin problems of sparsity testing and sparsity estimation are closely connected, we start by discussing the literature mostly related to the test version of our problem and then turn to the estimation version.

Signal detection.

The signal detection problem which amounts to testing whether $\theta=0$ is a special instance of the sparsity testing problem (corresponding to $k_{0}=0$). Signal detection in the Gaussian vector model has been extensively studied  [25, 3, 17, 14] in the last fifteen years and is now well understood. For instance, it has been established in [14] that the minimax separation distance $\rho_{\gamma}^{*}[0,\Delta]$ satisfies

[TABLE]

where $f(\Delta,n)\asymp_{\gamma}g(\Delta,n)$ means that there exist positive constants $c_{\gamma}$ and $c^{\prime}_{\gamma}$ (possibly depending on $\gamma$) such that $f(\Delta,n)\leq c_{\gamma}g(\Delta,n)\leq c^{\prime}_{\gamma}f(\Delta,n)$. Besides, some tests are able to simultaneously achieve the above separation distances for all positive $\Delta$.

Looking more closely at the above equation, one can distinguish two main regimes for this problem depending on the sparsity $\Delta$ of the alternative: the sparse case ($\Delta\leq\sqrt{n}$) and the dense case ($\Delta>\sqrt{n}$). In the sparse case, $\rho_{\gamma}^{*2}[0,\Delta]$ is of order $\Delta\log(1+n/\Delta^{2})$. This entails that it is possible to detect sparse vectors $\theta$ whose non-zero values are of order $\sqrt{\log(n/\Delta^{2})}$. Known optimal tests such as the higher criticism test [17] or the one proposed in [14] amount to counting the number of values $|Y_{i}|$ that are larger than $t$ and to compare this number to what is expected under the null hypothesis. Doing this simultaneously for a wide range of $t$ leads to near-optimal performances simultaneously for all $\Delta\in[1,\sqrt{n}]$. In the dense case ($\Delta\geq\sqrt{n}$), the situation is qualitatively different as the square minimax separation distance $\rho_{\gamma}^{*2}[0,\Delta]$ is of order $\sqrt{n}$. A near-optimal test, proposed in e.g.  [3], is based on the statistic $\|Y\|_{2}^{2}/\sigma^{2}$, which, under the null, follows a $\chi^{2}$ distribution with $n$ degrees of freedom and, under the alternative, follows a non-central $\chi^{2}$ distribution with non-centrality parameter $\|\theta\|_{2}^{2}$.

Composite-composite testing problems and functional estimation.

For the signal detection problem ($k_{0}=0$), the null hypothesis is simple whereas for the general case $k_{0}>0$, the null hypothesis is composite, thereby making the analysis of the problem more challenging. Although we are not aware of any general treatment of this kind of problem in the literature (and we are also not aware of the treatment of our specific problem in the literature), some partial results and methods may be derived in our setting from prior approach on related problems.

Minimax analysis of composite-composite testing problems has, up to our knowledge, been tackled in a few work [28, 4, 16, 12]. Some functional estimation problems, whose goal is to infer $f(\theta)$ for a given function $f$, are also related to some composite-composite testing problems. In fact, some work on functional problems [32, 19, 10, 8, 7] and adaptive confidence regions (e.g.  [9, 11, 23, 37]) have lead to progress in the understanding of such testing problems.

To be more specific on the challenge of composite-composite problems, let us describe a natural approach called ”infimum testing” [20]. Consider a signal detection test based on the statistic $S(.)$, that rejects the null hypothesis $H_{k_{0}}$ with $k_{0}=0$ for large values of $S(Y)$. The corresponding infimum test for the composite-composite testing problem (4), is a test rejecting $H_{k_{0}}$ for large values of $S_{\inf}:=\inf_{u\in\mathbb{B}_{0}[k_{0}]}S(Y-u)$. Indeed, there exists, under the null hypothesis $H_{k_{0}}$, some $u$ such that the expectation of $Y-u$ is zero. As one may expect, considering this infimum over all possible parameters in the null hypothesis is not priceless and the separation distance $\rho_{\gamma}[T_{k_{0}};k_{0},\Delta]$ of the corresponding infimum test $T_{k_{0}}$ may depend on the complexity of the null hypothesis. Conversely, simple inclusion arguments that will be recalled in our proofs entail that the composite problem is at least as difficult as the signal detection problem, that is $\rho^{*}_{\gamma}[k_{0},\Delta]$ is at least of the order of $\rho^{*}_{\gamma}[0,\Delta]$. The main challenge is therefore to decipher whether $\rho^{*}_{\gamma}[k_{0},\Delta]$ is indeed of order $\rho^{*}_{\gamma}[0,\Delta]$ or if it is larger than that and really depends on $k_{0}$. In other words, we seek to understand how the complexity of the null hypothesis influences the difficulty of the testing problem.

Sparsity estimation.

Closer to our setting, Cai, Jin and Low [8] study the problem of estimating $\|\theta\|_{0}$ for sparse vectors $\theta$ such that $\|\theta\|_{0}\leq\sqrt{n}$. They consider a Bayesian framework, where each component $\theta_{i}$ is drawn independently from a two points mixture distribution $(1-\eta)\delta_{0}+\eta\delta_{a}$ for some unknown $a>0$ ($\delta_{x}$ denotes the Dirac measure at $x$). The goal is then to estimate $\eta=\operatorname{\mathbb{E}}[\|\theta\|_{0}]/n$. Relying on the tail distribution of $Y$, they introduce an estimator $\widehat{\eta}$ that satisfies $\widehat{\eta}\leq\eta$ with high probability and such that the risk $\operatorname{\mathbb{E}}[|1-\widehat{\eta}/\eta|]$ is as small as possible. In [26], Jin introduced a class of estimators of $\theta$ based on the empirical characteristic function of $Y$ to handle the denser case $\|\theta\|_{0}\geq\sqrt{n}$. Later, these procedures have been extended [27, 7] to allow for unknown noise level $\sigma$ and even unknown mean in the more general model $Y_{i}=u+\theta_{i}+\epsilon_{i}$, where $u$ is unknown. Again, in a Bayesian framework where all $\theta_{i}$’s follow the same mixture distribution $(1-\eta)\delta_{0}+\eta\pi$ for some smooth density $\pi$, their estimator $\widehat{\eta}$ is proved to achieve an optimal minimax rate.

In multiple testing, estimating the number of false hypotheses has a longer history. Rephrased in the Gaussian vector model, multiple hypotheses testing amounts to test simultaneously whether each $\theta_{i}$ is zero or not. Hence, estimating the number of false hypotheses is equivalent to sparsity estimation. Nevertheless, most work on this field (e.g.  [38, 36, 39, 13, 31]) consider a more general setting where each $Y_{i}$ follows a mixture of a normal distribution and some unknown distribution that stochastically dominates the normal distribution. Hence, the methods and results are not directly comparable to ours.

1.4 Further description of our results

We now discuss in more details our three contributions mentionned in Section 1.2.

Sparsity testing for known $\sigma$.

Table 1 summarizes the squared minimax separation distances $\rho^{*2}_{\gamma}[k_{0},\Delta]$. Interestingly, for $k_{0}\leq\sqrt{n}$, the minimax separation distance is the same as for signal detection ($k_{0}=0$). In contrast, for more complex null hypotheses ($k_{0}\geq\sqrt{n}$), the complexity of the null hypothesis comes into play. For instance, when $\Delta\geq k_{0}\geq\sqrt{n}$, then $\rho^{*2}_{\gamma}[k_{0},\Delta]$ is of order $k_{0}/[\log(1+\tfrac{k_{0}}{\sqrt{n}})]$. This is smaller by a polylog multiplicative term than what can be obtained by infimum tests and we have to rely on really different statistics. In fact, our minimax adaptive procedure is a combination of three tests. The first one is an adaptation of the the higher criticism test introduced in [17]. The second one relies on the empirical characteristic function of $Y$ and borrows ideas from [26]. The third statistic is novel and relies on deconvolution ideas. As for the lower bounds of the minimax separation distances for large $k_{0}$, the proof ideas are more involved than for signal detection [3] and make use of the moment matching techniques introduced in [32] and later refined in [28, 10].

Sparsity testing for unknown $\sigma$.

The results discussed above hold under the restrictive assumption that the noise level $\sigma$ is known. For unknown $\sigma$, the situation is qualitatively different (see Table 2). As a first step, we study the signal detection problem ($k_{0}=0$) for which only partial results had been established. For sparse alternatives ($\Delta\leq\sqrt{n}$), one can plug an estimator of $\sigma$ in the signal detection statistic so that the minimax separation distance $\rho^{*}_{\gamma,\mathrm{var}}(0,\Delta)$ for unknown variance (defined in (41)) is the same as $\rho^{*}_{\gamma}(0,\Delta)$. However, for $\Delta$ larger than $\sqrt{n}$ and much smaller than $n$, one cannot simply plug a variance estimator and new test statistics are required. The squared separation distance $\rho^{*2}_{\gamma,\mathrm{var}}(0,\Delta)$ is of order $\sqrt{\Delta n^{1/2}}$ whereas $\rho^{*2}_{\gamma}(0,\Delta)$ is only of order $\sqrt{n}$. In the really dense case where $\Delta$ is proportional to $n$, we establish that the separation distance $\rho^{*2}_{\gamma,\mathrm{var}}(0,\Delta)$ is even larger. Turning to the general case $k_{0}>0$, we establish that $\rho^{*}_{\gamma,\mathrm{var}}(k_{0},\Delta)$ is larger than its counterpart for known $\sigma$ for all $\Delta\geq\sqrt{n}\vee k_{0}$. In comparison to the known variance case, one cannot simply accommodate the adaptive test by estimating the noise level. In fact, the minimax adaptive test in this new setting is based on quite different statistics.

Sparsity estimation.

Let us first verbalize the desirable properties of a good estimator of $\|\theta\|_{0}$. The functional $\|\theta\|_{0}$ is not continuous with respect to $\theta$. Consider a one-sparse vector $\theta$ (with one large non-zero component) and a perturbation $\theta^{\prime}$ of $\theta$ whose components are all nonzero but are arbitrarily small. As the distribution $\operatorname{\mathbb{P}}_{\theta,\sigma}$ is close to $\operatorname{\mathbb{P}}_{\theta^{\prime},\sigma}$, the estimator $\widehat{k}$ will follow almost the same distribution for both parameters. It is obviously preferable for $\widehat{k}$ to be concentrated around one under $\operatorname{\mathbb{P}}_{\theta^{\prime},\sigma}$ than around $n$ under $\operatorname{\mathbb{P}}_{\theta,\sigma}$. In other words, a good estimator $\widehat{k}$ should have a small overestimation probability. Besides, a good estimator $\widehat{k}$ should be larger than any fixed $q$, as soon as the distance of $\theta$ to the collection $\mathbb{B}_{0}[q]$ is large enough.

To formalize the above intuition, let us consider the multiple testing problems with all hypotheses $(H_{q})$, for $q=0,\ldots,n$ where $H_{q}$ is defined in (4). Then, the set of true hypotheses is exactly $\{H_{q},\ q\geq\|\theta\|_{0}\}$. Similarly, an estimator $\widehat{k}$ of $\|\theta\|_{0}$ can be interpreted as a multiple testing procedure rejecting all hypotheses $H_{q}$ with $q<\widehat{k}$ and accepting all hypotheses $H_{q}$ with $q\geq\widehat{k}$. Conversely, one can build an estimator of $\|\theta\|_{0}$ from any multiple testing procedure. Building on this correspondence between complexity tests and complexity estimation, we first construct a multiple sparsity testing procedures. Although the minimax optimality of multiple testing procedures is difficult to assess (but see [18]), we are able to prove that our procedure is simultaneously minimax for all single hypotheses $H_{q}$. Then, the corresponding estimator $\widehat{k}$ satisfies, with high probability, the three following properties

1. (a)

$\widehat{k}\leq\|\theta\|_{0}$, which is equivalent to $\theta_{(\widehat{k})}\neq 0$ (Here $\theta_{(i)}$ stands for the $i$-th largest entry of $\theta$ in absolute value333Consequently, we have $|\theta_{(1)}|\geq|\theta_{(2)}|\geq\ldots\geq|\theta_{(n)}|$. with the convention $\theta_{(0)}=+\infty$). 2. (b)

For all $q=1,\ldots,n-\widehat{k}$, $|\theta_{(\widehat{k}+q)}|\leq c\psi_{\hat{k},q}$, where $c$ is a numerical constant and the function $\psi_{\widehat{k},q}$ is defined in  (23). In other words, we can certify, that even if $\widehat{k}$ is possibly smaller than $\|\theta\|_{0}$, each of its remaining $(\|\theta\|_{0}-\widehat{k})$ non-zero components are small enough. 3. (c)

$d_{2}(\theta,\mathbb{B}_{0}[\widehat{k}])\leq c^{\prime}\rho_{\gamma}^{*}[\widehat{k},\|\theta\|_{0}-\widehat{k}]$, where $c^{\prime}$ is a numerical constant and $\gamma$ is fixed. In other words, $\theta$ is close in $l_{2}$ distance to the collection of $\widehat{k}$-sparse vectors.

Note that both properties (a) and (b) produce data-driven certificates for all $\theta_{(\widehat{k}+q)}$, $q\geq 0$ in the sense that corresponding bounds are explicit. Besides, the three above properties are valid for all $\theta\in\mathbb{R}^{n}$, whereas previous work [8, 27, 7] only considered specific classes $\theta$ by assuming for instance that the $\theta_{i}$’s are sampled according to a mixture of a Dirac at [math] and a smooth distribution. For a given $\theta$, one can invert the inequalities in conditions (b) and (c) to obtain a bound for $|\widehat{k}-\|\theta\|_{0}|$. Finally, both conditions (b) and (c) are optimal from a minimax perspective defined in Section 3.

1.5 Notation and organization of the paper

Although some of the notation have already been introduced, we gather them here to ease the reading. Given a vector $u\in\mathbb{R}^{n}$ and $p\geq 1$, we denote $\|u\|_{p}^{p}:=(\sum_{i}|u_{i}|^{p})^{1/p}$ its $l_{p}$ norm. Also, $\|u\|_{\infty}:=\max_{i}|u_{i}|$ stands for its $l_{\infty}$ norm and $\|u\|_{0}=\sum_{i}{\mathbf{1}}_{u_{i}\neq 0}$ its $l_{0}$ function. In the sequel, $\phi(.)$ stands for the density of a standard normal variable, and $\Phi(.)$ for its survival function. Also $\mathcal{N}(x,\sigma^{2})$ stands for the normal distribution with mean $x$ and variance $\sigma^{2}$. Given $x\in\mathbb{R}$, we write as usual $\lfloor x\rfloor$ for the integer part of $x$ and $\lceil x\rceil$ for the rounding to the upper integer, and $(x)_{+}:=\max(x,0)$. Also $[n]$ is short for the set $\{1,\ldots,n\}$. For any $i\in[n]$, $\theta_{(i)}$ stands for the $i$-th largest entry of $\theta$ in absolute value. In other words, one has $|\theta_{(1)}|\geq|\theta_{(2)}|\geq\ldots\geq|\theta_{(n)}|$.

In the sequel, $c$, $c_{1}$, $\ldots$ denote positive universal constants that may change from line to line. We also denote $c_{\alpha}$, $c^{\prime}_{\beta}$,…, denote positive constants whose values may depend on $\alpha$ or $\beta$.

When $Y$ is distributed according to the model (1), we write $\operatorname{\mathbb{P}}_{\theta,\sigma}$ for the distribution of $Y$. As $\sigma$ is fixed and supposed to be known in Sections 2 and 3, we drop the dependency on $\sigma$ in these two sections and simply write $\operatorname{\mathbb{P}}_{\theta}$.

In Section 2, we describe our model testing results when the variance of the noise is known, presenting both upper and lower bounds. In Section 3, we detail how these testing results can be applied to the relevant problem of sparsity estimation. Section 4 is devoted to the unknown variance case. Finally, remaining results and all the proofs are postponed the Appendix.

2 Sparsity testing with known variance

2.1 Minimax lower bound

In this section, we consider the the sparsity testing problem (4) in a setting when the noise variance $\sigma^{2}$ is known. The following theorem states a lower bound on the minimax separation distance $\rho^{*}_{\gamma}[k_{0},\Delta]$.

Theorem 1**.**

There exists a numerical constant $c>0$ such that the following holds. Consider any $\gamma\leq 0.5$. For any $k_{0}\leq\sqrt{n}$ and $\Delta\leq n-k_{0}$, we have

[TABLE]

For any $k_{0}>\sqrt{n}$, we have

[TABLE]

As proved in the next subsection, this lower bound turns out to be sharp. We shall precisely discuss these quantities later. Before this, we only give a glimpse of the different regimes unveiled by the above theorem.

Whenever $k_{0}\leq\sqrt{n}$, the lower bound on the minimax separation distance is the same as the signal detection minimax separation distance $\rho^{*}_{\gamma}[0,\Delta]$, see [3, 14]. In this regime, the size $k_{0}$ of the null hypothesis does not play a role in the separation distance. In fact, the proof of (8) is a consequence of known results for the signal detection problem. More precisely, we follow Le Cam’s method and choose a particular $\theta_{0}\in\mathbb{B}_{0}[k_{0}]$ and a prior distribution $\nu$ on the collection $\mathbb{B}_{0}[k_{0}+\Delta,k_{0},\rho]$. Let us write $\mathbb{Q}_{1}:=\int\mathbb{P}_{\theta}\nu(d\theta)$ the marginal distribution of $Y$ when $\theta$ is sampled according to $\nu$. Then, the risk $R(T;k_{0},\Delta,\rho)$ (5) of any test $T$ is larger than $1-\|\mathbb{P}_{\theta_{0}}-\mathbb{Q}_{1}\|_{TV}$ ($\|.\|_{TV}$ is the total variation distance). Since the total variation distance is dominated by the $\chi^{2}$ distance between probability distributions, it suffices to bound from above this $\chi^{2}$ distance.

For $k_{0}$ much larger than $\sqrt{n}$ and for $\Delta\geq k_{0}$, the lower bound (9) is of order k_{0}/\log\big{[}\tfrac{k_{0}}{\sqrt{n}}\big{]} - which is significantly larger than the signal detection rate $\rho^{*}_{\gamma}[0,\Delta]$. In this regime, the complexity of the null hypothesis $H_{k_{0}}$ has to be taken into account to obtain the right lower bound. Following an approach pioneered in [32], we build two product prior distributions $\mu_{0}^{\otimes n}$ and $\mu_{1}^{\otimes n}$ (almost) supported by $\mathbb{B}_{0}[k_{0}]$ and $\mathbb{B}_{0}[k_{0}+\Delta,k_{0},\rho]$ in such a way that the first moments of $\mu_{0}$ and $\mu_{1}$ are matching. Writing $\mathbb{Q}_{0}:=\int\mathbb{P}_{\theta}\mu_{0}^{\otimes n}(d\theta)$ and $\mathbb{Q}_{1}:=\int\mathbb{P}_{\theta}\mu_{1}^{\otimes n}(d\theta)$, we need to upper bound the $\chi^{2}$ distance between $\mathbb{Q}_{0}$ and $\mathbb{Q}_{1}$. It turns out that matching the moments of $\mu_{0}$ and $\mu_{1}$ enforces the $\chi^{2}$ distribution between $\mathbb{Q}_{0}$ and $\mathbb{Q}_{1}$ to be small enough. The main technical hurdle in the proof is the construction of the two measures $\mu_{0}$ and $\mu_{1}$ that maximize the the number of matching moments, while being supported respectively on the null and alternative hypothesis with $\rho$ as large as possible.

2.2 Minimax upper bound

In this subsection, we construct three tests that are most effective in three different situations: the Higher Criticism regime (large but few non-zero components), the Bulk regime (many but small non-zero components) and the Intermediary regime. Then, a combination of these three procedures is proved to achieve the minimax lower bounds of Theorem 1 and is even adaptive to the sparsity $k_{1}$. Throughout this subsection, we consider some fixed $\alpha$ and $\beta$ in $(0,1)$.

2.2.1 Higher Criticism Statistic

Let us adapt the Higher Criticism statistic introduced in [17] for signal detection. Recall that, for $t>0$, $\Phi(t)$ is the survival function of the standard normal distribution For any $t>0$, define

[TABLE]

the number of components larger (in absolute value) than $t$, $t^{HC}_{*,\alpha}:=\lceil\sqrt{2\log[4n/\alpha]}\rceil$ and the collection $\mathcal{T}_{\alpha}:=[t^{HC}_{*,\alpha}]$. Then, the test $T^{HC}_{\alpha,k_{0}}$ rejects the null hypothesis $H_{k_{0}}$, if either $N_{\sigma t^{HC}_{*,\alpha}}\geq k_{0}+1$ or for some $t\in\mathcal{T}_{\alpha}$,

[TABLE]

where

[TABLE]

Under the the null hypothesis $H_{k_{0}}$, $\theta$ contains at most $k_{0}$ non zero coefficients and $N_{\sigma t}-k_{0}$ is therefore stochastically dominated by a Binomial random variable with parameters $(n-k_{0},2\Phi(t))$. It then follows from Chebychev inequality that $N_{\sigma t}\leq k_{0}+2(n-k_{0})\Phi(t)+O_{p}(\sqrt{(n-k_{0})\Phi(t)})$. The specific choice of the tuning parameter $u^{HC}_{t,\alpha}$ allows to handle the multiplicity of the tests. In the specific case $k_{0}=0$ (signal detection), $T^{HC}_{\alpha,k_{0}}$ is analogous to the vanilla Higher Criticism test [17].

Proposition 1**.**

The size of the test $T^{HC}_{\alpha,k_{0}}$ is smaller of equal to $\alpha$. Besides, any $\theta\in\mathbb{R}^{n}$ such that

[TABLE]

belongs to the high probability rejection region of $T^{HC}_{\alpha,k_{0}}$, that is $\operatorname{\mathbb{P}}_{\theta}[T^{HC}_{\alpha,k_{0}}=1]\geq 1-\beta$.

In the specific case $k_{0}=0$, we recover the known behavior or the Higher Criticism statistic in the signal detection setting. The test $T^{HC}_{\alpha,k_{0}}$ is powerful when, for a given integer $q$, there are least $(k_{0}+q)$ coefficients larger than some threshold depending on $q$. For $q=1$, the threshold is of order $\sigma\sqrt{\log(n)}$, whereas for $q\geq\sqrt{n}\vee k_{0}$, the threshold is of order one. It will turn out that $T^{HC}_{\alpha,k_{0}}$ achieves the optimal separation $\rho^{*}_{\alpha+\beta}[k_{0},\Delta]$ when $\Delta\leq\sqrt{n^{1/2}k_{0}\vee n}$. However, the test $T^{HC}_{\alpha,k_{0}}$ does not manage to detect vectors $\theta$ containing many coefficients that are small in front of one. This is why we follow another approach in this regime.

2.2.2 Detecting the signal in the bulk distribution

When there are many small coefficients, we rely on the empirical characteristic functions of $Y$ following an approach introduced in [26]. Given $s>0$, define the function

[TABLE]

and the test statistic $Z(s)$

[TABLE]

Let us describe the intuition behind this statistic using a population approach. Denoting $\overline{\varphi}_{n}(s)$ the empirical characteristic function and $\overline{\varphi}(s)$ its expectation

[TABLE]

one can derive the expectation of $Z(s)$

[TABLE]

with the convention $(1-\cos(x))/x^{2}=1/2$ for $x=0$. Since, for all $x$, $\cos(x)\in[1-x^{2}/2,1]$, one may easily show (see the proof of Proposition 2 for details) that $\operatorname{\mathbb{E}}_{\theta}[Z(s)]\leq\|\theta\|_{0}$. Under the null, this expectation is therefore smaller or equal to $k_{0}$. Besides, a Taylor development of the $\cos$ function around [math] ensures that $1-2\frac{1-\cos(sx)}{(sx)^{2}}=\frac{1}{12}(sx)^{2}+o(s^{2}x^{2})$. If, under the alternative, there are so many small coefficients $|\theta_{i}|$ that the corresponding sum $\sum_{i}\theta_{i}^{2}s^{2}/\sigma^{2}$ is large in front of $k_{0}$, then, at least in expectation, $Z(s)$ is larger than under the null.

Remark: Rewriting the statistic $Z(s)/n=1-\int_{-1}^{1}(1-|\xi|)e^{s^{2}\xi^{2}/2}\overline{\varphi}_{n}(s\xi/\sigma)d\xi$, one observes that the empirical characteristic function is multiplied by the function $(1-|\xi|)$ before integration. In [26], Jin also suggests other statistics such as $\int_{-1}^{1}e^{s^{2}\xi^{2}/2}\overline{\varphi}_{n}(s\xi/\sigma)d\xi$ or the deconvolution estimator $e^{s^{2}/2}\overline{\varphi}_{n}(s/\sigma)$. However, these two statistics turn out to be suboptimal in our setting.

In practice, we set $s_{k_{0}}:=\sqrt{\log(ek^{2}_{0}/n)}\vee 1$ and we define the test $T^{B}_{\alpha,k_{0}}$ rejecting the null hypothesis when

[TABLE]

Proposition 2**.**

There exist three positive constants $c_{\alpha,\beta},c^{\prime}_{\alpha,\beta},c^{{}^{\prime\prime}}_{\alpha,\beta}$ such that the following holds. The type I error probability of $T^{B}_{\alpha,k_{0}}$ is smaller or equal to $\alpha$. Besides, any $\theta\in\mathbb{R}^{n}$ satisfying any of the two following conditions

[TABLE]

belongs to the high probability rejection region of $T^{B}_{\alpha,k_{0}}$, that is $\operatorname{\mathbb{P}}_{\theta}[T^{B}_{\alpha,k_{0}}=1]\geq 1-\beta$.

The above proposition provides two sufficient condition for $T^{B}_{\alpha,k_{0}}$ to be powerful. The second condition (19) formalizes the above discussion for the population version of the statistic: when the squared $l_{2}$ norm of the restriction of $\theta$ to its small coefficients is larger in front of $\sigma\frac{k_{0}}{\log(1+k_{0}/\sqrt{n})}$, then the test is powerful. Condition (18) ensures that the test is also powerful when there are more than $k_{0}+q$ coefficients larger than some threshold depending on $q$. In comparison to the Higher Cristicism test, Condition (18) is effective for large $q$ (many non-zero coefficients), but these coefficients can be much smaller than one.

2.2.3 Intermediary regimes

A combination of the two previous tests covers the extreme regimes for the sparsity testing problem: a few large coefficients (Higher Criticism) and many small coefficients (Bulk). Unfortunately, they turn out to be suboptimal in intermediate regimes ie.  for any parameters in between. This is why we have to devise a third test. In this subsection we aim at discovering intermediary signals whose signature is neither in the bulk of the empirical distribution of $(Y_{i})$ nor in its extreme values. This problem will only reveal to be relevant for large $k_{0}$ and we assume henceforth that $k_{0}\geq 20\sqrt{n}$.

Given two tuning parameters $r$ and $l$, define the function

[TABLE]

and the statistic

[TABLE]

In order to get a grasp this statistic let us consider the expectation of $\eta_{r,w}(X)$ for $X\sim\mathcal{N}(x,1)$. Simple computations (see (107) in the proof of Proposition 3) lead to

[TABLE]

which, for large $r$, is of order $\int_{\mathbb{R}}\phi(\xi)\cos(\xi x\frac{w}{r})d\xi=\exp(-x^{2}\tfrac{w^{2}}{2r^{2}})$. As a consequence, $\operatorname{\mathbb{E}}_{\theta}[V(r,w)]$ approximates the function $\|\theta\|_{0}$ at an exponential rate. In contrast, the population version of $Z(s)$ (15) only approximates the function $\|\theta\|_{0}$ at a quadratic rate. Unfortunately, the variance $V(r,w)$ is quite large which prevents us to take $w/r$ as large as $s_{k_{0}}$ as in the previous test.

The test $T^{I}_{\alpha,k_{0}}$ is an aggregation of multiple tests based on the statistics $V(r,w)$ for different tuning parameters $r$ and $w$. Define $l_{k_{0}}:=\lceil(k_{0}\sqrt{n})^{1/2}\rceil$ and the dyadic collection $\mathcal{L}_{k_{0}}=\{l_{k_{0}},2l_{k_{0}},4l_{k_{0}},\ldots,l_{\max}\}$ where $l_{\max}:=2^{\lfloor\log_{2}(k_{0}/l_{k_{0}})\rfloor}l_{k_{0}}/4\leq k_{0}/4$. Note that $\mathcal{L}_{k_{0}}$ is not empty if $k_{0}\geq 20\sqrt{n}$ and $n$ is large enough. Given any $l\in\mathcal{L}_{k_{0}}$, define

[TABLE]

Then, the test $T^{I}_{\alpha,k_{0}}$ rejects the null hypothesis if, for some $l\in\mathcal{L}_{k_{0}}$,

[TABLE]

where $\log_{2}$ is the binary logarithm.

Proposition 3**.**

There exists four positive constants $c,c_{\alpha,\beta},c^{{}^{\prime}}_{\alpha,\beta},c^{{}^{\prime\prime}}_{\alpha,\beta}$ such that the following holds. Assume that $k_{0}\geq 20\sqrt{n}$ and $n\geq c$. The type I error probability of $T^{I}_{\alpha,k_{0}}$ is smaller of equal to $\alpha$. If $k_{0}\geq c_{\alpha,\beta}\sqrt{n}$, any $\theta\in\mathbb{R}^{n}$ satisfying

[TABLE]

belongs to the high probability rejection region of $T^{I}_{\alpha,k_{0}}$, that is $\operatorname{\mathbb{P}}_{\theta}[T^{I}_{\alpha,k_{0}}=1]\geq 1-\beta$.

2.3 Combination of the tests

For any integer $q\in[n-k_{0}]$, define $\psi_{k_{0},q}>0$ by

[TABLE]

Let $T^{C}_{\alpha,k_{0}}$ denote the aggregation of the three previous tests. We take T^{C}_{\alpha,k_{0}}:=\max\big{(}T^{HC}_{\alpha/3,k_{0}},T^{B}_{\alpha/3,k_{0}},T^{I}_{\alpha/3,k_{0}}\Big{)}, if $k_{0}\geq 20\sqrt{n}$ and $T^{C}_{\alpha,k_{0}}:=\max(T^{HC}_{\alpha/2,k_{0}},T^{B}_{\alpha/2,k_{0}})$ else. The following result holds.

Corollary 1**.**

There exist three constants $c$, $c_{\alpha,\beta}$, and $c^{\prime}_{\alpha,\beta}$ such that the following holds for $n\geq c$. The type I error probability of $T^{C}_{\alpha,k_{0}}$ is smaller than $\alpha$. Besides, $\operatorname{\mathbb{P}}_{\theta}[T^{C}_{\alpha,k_{0}}=1]\geq 1-\beta$ for any vector $\theta$ such that

[TABLE]

Also, $\operatorname{\mathbb{P}}_{\theta}[T^{C}_{\alpha,k_{0}}=1]\geq 1-\beta$ for any vector $\theta$ satisfying,

[TABLE]

In view of Theorem 1 and (25) in Corollary 1, it holds that $\rho^{*}_{\alpha+\beta}[k_{0},\Delta]\asymp_{\gamma}\sigma^{2}\Delta\psi^{2}_{k_{0},\delta}$. Besides, the test $T^{C}_{\alpha,k_{0}}$ simultaneously achieves (up to multiplicative constants) these minimax separation distances over all $\Delta\in[n-k_{0}]$. Condition (24) provides a complementary characterization of $T^{C}_{\alpha,k_{0}}$ power function. This bound will be central for sparsity estimation in the next section.

To conclude this section, we summarize the results on the testing separation distance $\rho_{\gamma}^{*2}[k_{0},\Delta]$ as depicted in Table 1 in the introduction. For $k_{0}\leq\sqrt{n}$, $\rho_{\gamma}^{*}[k_{0},\Delta]$ is of same order as the signal detection separation distance $\rho_{\gamma}^{*}[0,\Delta]$. For $k_{0}>\sqrt{n}$, the minimax-optimal separation distance $\rho_{\gamma}^{*}[k_{0},\Delta]$ becomes significantly larger than the signal detection separation distance. The complexity of the null hypothesis plays an important role in $\rho_{\gamma}^{*}[k_{0},\Delta]$. For instance, when $k_{0}=n^{\zeta}$ with $\zeta>1/2$ and for $\Delta\geq k_{0}$, $\rho_{\gamma}^{*2}[k_{0},\Delta]$ is of order $k_{0}/\log(n)$. Besides, for $k_{0}$ between $\sqrt{n^{1/2}k_{0}}$ and $k_{0}$, there is smooth transition from squared separation distances of order $\Delta\log(n)$ to $\Delta/\log(n)$.

3 Sparsity estimation

Given an observation $Y$, our goal is now to estimate the number $\|\theta\|_{0}$ of non-zero components of $\theta$. As explained in the introduction, we rephrase this estimation problem as a multiple testing problem. Let $\mathcal{H}=(H_{k})_{k=0,\ldots,n}$ denote the nested collection of all hypotheses $H_{k}$ (4). For a parameter $\theta$, the set of true hypotheses $\mathcal{T}(\theta)$ is the collection $\{H_{k},k\geq\|\theta\|_{0}\}$ and the set of false hypotheses $\mathcal{R}(\theta)$ is the collection $\{H_{k},k<\|\theta\|_{0}\}$. A multiple hypothesis test is a measurable collection $\widehat{\mathcal{R}}\subset\mathcal{R}$.

Let us make explicit the connection between these two problems. Given an estimator $\widehat{k}$ of $\|\theta\|_{0}$, taking $\widehat{\mathcal{R}}=\{H_{k},k<\widehat{k}\}$ defines a multiple test. Conversely, consider a multiple test $\widehat{\mathcal{R}}$. Then, one may define the estimator $\widehat{k}=1+\max\{k:H_{k}\in\widehat{\mathcal{R}}\}$. In our framework, a closed test $\widehat{\mathcal{R}}$ is a test that satisfies the property “$H^{\prime}\subset H$ and $H\subset\widehat{\mathcal{R}}$ implies $H^{\prime}\subset\widehat{\mathcal{R}}$” (see e.g.  [18]). It follows from the above constructions that sparsity estimators $\widehat{k}$ are in one to one correspondence with closed testing procedures.

The above correspondence leads us (i) to build estimators $\widehat{k}$ that rely on the test statistics introduced in the previous section and (ii) to evaluate the performances of $\widehat{k}$ in terms of separation distances of a multiples testing procedure.

3.1 From single tests to multiple tests

Fix some $\alpha\in(0,1)$. As in the previous section, our estimator $\widehat{k}$ defined by

[TABLE]

is based on a combination of three statistics respectively corresponding to tests of the form $T^{HC}_{\alpha,k_{0}}$, $T^{B}_{\alpha,k_{0}}$ and $T^{I}_{\alpha,k_{0}}$. However, contrary to these tests, we have to deal with many null hypotheses.

Construction of $\widehat{k}_{HC}$

Let $t_{*}:=t_{*,\alpha/3}^{HC}$ where $t_{*,\alpha/3}^{HC}$ is defined in Section 2.2.1 and write $\mathcal{T}=[t_{*}]$. Define the Higher-Criticism estimator of $\|\theta\|_{0}$ by

[TABLE]

where $N_{t}$ and $u^{HC}_{t,\alpha}$ are introduced in Section 2.2.1. Note that $\widehat{k}_{HC}$ is quite similar to the estimator of Meinshausen and Rice [35] developed in a mixture model setting . Let us explain the rationale between this estimator. First, $N_{\sigma t_{*}}$ is the number of coordinates of $Y$ larger than $t_{*}$ (in absolute value). Deviation inequalities of the normal distribution enforce that, with high probability, each of these coordinates corresponds to a non-zero component of $\theta$. For $t\in\mathcal{T}$, Bernstein’s inequality enforces that, with high probability, there are less than $2(n-\|\theta\|_{0})\Phi(t)+u^{HC}_{t,\alpha/3}$ components of $Y$ larger than $\sigma t$ in absolute values that correspond to null components $\theta_{i}$. As a consequence, $N_{\sigma t}-2n\Phi(t)-u^{HC}_{t,\alpha/3}$ is, with high probability, a lower bound of the number of non-zero coordinates of $\theta$.

Construction of $\widehat{k}_{B}$ and $\widehat{k}_{I}$

Following the intuition explained in the introduction, it would be tempting to define $\widehat{k}_{B}-1$ as the largest $q\in[n]$ such that the test $T^{B}_{\alpha_{q},q}$ (with some suitable tuning parameters $\alpha_{q}$) rejects the null. However, this simple strategy leads to a logarithmic loss in comparison to the optimal testing separation rate. As explained in Sections 2.2.2 and 2.2.3, the the statistics $Z(s)$ and $V(r,w)$ involved in the tests $T^{B}_{\alpha,k_{0}}$ and $T^{I}_{\alpha,k_{0}}$ can be interpreted as (possibly biased) estimators of $\|\theta\|_{0}$. The bias and the variance of these estimators depends on choice of the tuning parameters $s$, $r$ and $w$. For instance, for a large value of $s$, the variance $Z(s)$ is higher but $\operatorname{\mathbb{E}}_{\theta}[Z(s)]$ is close to $\|\theta\|_{0}$ (see Section 2.2.2). This is why we shall compute these statistics for a large collection of tuning parameters.

Introducing $k_{\min}:=\lceil\sqrt{n}\rceil$, we shall consider the dyadic collection $\mathcal{K}_{0}:=\{k_{\min},2k_{\min},\ldots,k_{\max}\}$, where $k_{\max}\in(n/2;n]$. In order to calibrate this large collection of statistics, we have to adjust the thresholds $u^{B}_{k_{0},\alpha}$ and $u^{I}_{k_{0},l,\alpha}$ of the statistics. For any $k_{0}\in\mathcal{K}_{0}$, denote $\alpha_{k_{0}}:=2\alpha([1+\log_{2}(\tfrac{k_{0}}{k_{\min}})]^{2}\pi^{2})^{-1}$ so that $\sum_{k_{0}\in\mathcal{K}_{0}}\alpha_{k_{0}}\leq\alpha/3$. Equipped with this notation, we define the Bulk and Intermediary estimators of $\|\theta\|_{0}$ as follows

[TABLE]

where $Z(s)$, $V(r,w)$, $u_{k_{0},\alpha}^{B}$ and $u_{k_{0},l,\alpha}^{I}$ are introduced in Sections 2.2.2 and 2.2.3.

Remark. The number of statistics required to compute $\widehat{k}$ is of order $\log^{2}(n)$.

3.2 Optimal sparsity estimation rates

Theorem 2**.**

Fix any $\beta\in(0,1)$. There exists two positive constants $c_{\alpha,\beta}$ and $c^{\prime}_{\alpha,\beta}$ such that the following hold for any $\theta\in\mathbb{R}^{n}$. With high probability, $\widehat{k}$ does not overestimate the number of non-zero components,

[TABLE]

With probability larger than $1-\beta$, the vector $\theta$ contains no more than $\widehat{k}$ large coefficients in the sense that

[TABLE]

and

[TABLE]

where the sequence $\psi$ is defined in Equation (23).

As a consequence, outside an event of probability smaller than $\alpha+\beta$, we have $\widehat{k}\leq\|\theta\|_{0}$ and $\theta$ is so close to $\mathbb{B}_{0}[\widehat{k}]$ that is is impossible to reliably decipher whether $\theta\in\mathbb{B}_{0}[\widehat{k}]$ or not. Alternatively, Theorem 2 provides the following data-driven certificate: with high probability and simultaneously for all $q\geq 1$, there are no more than $\widehat{k}+q$ coefficients larger (up to constants) than $\psi_{\widehat{k},q}$.

Below, we state two straightforward corollaries of Theorem 2 providing alternative interpretations of the result. Recall the multiple testing procedure $\widehat{\mathcal{R}}$ derived from $\widehat{k}$.

Corollary 2**.**

The Family-wise error rate (FWER) of the procedure $\widehat{\mathcal{R}}$ is controlled at level $\alpha$:

[TABLE]

Given $\beta\in(0,1)$, there exists a constant $c_{\alpha,\beta}$ such that the following holds for all $\theta\in\mathbb{R}^{n}$. With probability larger than $1-\beta$, $\widehat{\mathcal{R}}$ contains all hypotheses $H_{k}$ such that

[TABLE]

In view of Section 2, the multiple testing procedure $\widehat{\mathcal{R}}$ simultaneously performs as well as any minimax adaptive single test of the hypothesis $H_{k_{0}}$ for a given $k_{0}=0,\ldots,n-1$. In other words, the multiplicity of the hypotheses does not induce any loss.

For a given $\theta$, we can easily ”invert“ the conditions (31) and (32) to control the error $|\widehat{k}-\|\theta\|_{0}|$.

Corollary 3**.**

There exists a positive constant $c_{\alpha,\beta}$ such that the following holds. For any $\theta\in\mathbb{R}^{n}$, the sparsity estimator satisfies the three following properties

[TABLE]

outside an event of probability smaller than $\alpha+\beta$. In the above equations, we choose the convention $\min\{\emptyset\}=\infty$ and $\max\{\emptyset\}=-\infty$.

Conversely, it is not possible to improve the bounds (34) and (35).

Corollary 4**.**

There exists a positive constant $c^{\prime}_{\alpha,\beta}$ such that the following holds. Fix any integers $q>0$ and $k>0$ such that $k+q\leq n$. No estimator $\tilde{k}$ can satisfy simultaneously $\inf_{\theta\in\mathbb{B}_{0}[k]}\operatorname{\mathbb{P}}_{\theta}[\tilde{k}\leq k]\geq 1-\alpha$ and at least one of the two following properties

[TABLE]

For any fixed $(r,q)$, if we replace $\psi^{2}_{r,q}$ in (34) by $\tfrac{c^{\prime}_{\alpha,\beta}}{c_{\alpha,\beta}}\psi^{2}_{r,q}$, then (33) cannot hold together with (34) on an event of large probability. The same optimality results holds for (35).

To better grasp the implication of (34), let us consider a toy example for which $\|\theta\|_{0}=n^{\gamma}$ for some $\gamma\in(0,1)$ and given $\Delta\in[1,\ldots,\|\theta\|_{0}]$, we define $m^{2}_{\Delta}=\frac{1}{\Delta}\sum_{j=1}^{\Delta}\theta_{(\|\theta\|_{0}+1-j)}^{2}$ the mean square of the $\Delta$ smallest non-zero values of $\theta$. Note that $m_{\Delta}$ is a non-decreasing function of $\Delta$. It corresponds to the typical value of the $\Delta$ smallest non-zero components of $\theta$. Depending on the behavior of $m_{\Delta}$ we may bound the error of the estimator of $\|\theta\|_{0}$. First, if $m_{1}$ is large in front $\sqrt{\log(n)}$, then we have $\widehat{k}=\|\theta\|_{1}$ with high probability. Then, we consider two subcases:

1. (i)

$\gamma\in(0,1/2)$. Take $\Delta=n^{\zeta}$ with $\zeta\in(0,\gamma]$.

[TABLE]

Conversely, if $m_{\|\theta\|_{0}}\leq c^{\prime}_{\alpha,\beta}\sigma\sqrt{(1/2-\gamma)\log(n)}$, then it is impossible to distinguish $\theta$ from [math]. As a consequence, the relative estimation precision is mainly driven by the proportion of non-zero components that are large in front of $\sigma\sqrt{\log(n)}$. 2. (ii)

$\gamma\in(1/2,1)$. Here, the situation is more intricate:

1. (a)

$\Delta=n^{\zeta}$ with $\zeta\in(0,\gamma)$.

[TABLE]

In that case, all non-zero components of $\theta$ except a polynomially small proportion of them are larger than $\sigma\sqrt{\log(n)}$ and the relative estimation error $\frac{|\|\theta\|_{0}-\widehat{k}|}{\|\theta\|_{0}}$ converges polynomially fast to zero. 2. (b)

$\Delta=\frac{\|\theta\|_{0}}{u_{n}}$ with $u_{n}\to\infty$ and $u_{n}n^{-\zeta}\to 0$ for all $\zeta>0$ .

[TABLE]

For concreteness, fix $u_{n}=\log^{\zeta}(n)$ with $\zeta>0$. the relative convergence rate is of order $\log^{-\zeta}(n)$ if all non-zero components of $\theta$ except a proportion $u^{-1}_{n}$ of them are larger than $\sigma\zeta\tfrac{\log\log(n)}{\sqrt{\log(n)}}$. 3. (c)

$\Delta=\zeta\|\theta\|_{0}$ with some $\zeta\in(0,1)$. If $m_{\Delta}\geq c_{\alpha,\beta}\sigma\frac{\log(1/\zeta)}{\sqrt{\gamma\log(n)}}$, then $\frac{\|\theta\|_{0}-\widehat{k}}{\|\theta\|_{0}}\leq(1-\zeta)$. In that setting, a fixed proportion of non-zero coefficients are larger than $\sigma\frac{1}{\sqrt{\log(n)}}$. One is able to estimate $\|\theta\|_{0}$ up to a constant multiplicative factor. 4. (d)

$\Delta=\|\theta\|_{0}(1-\log^{-\zeta}(n))$ with $\zeta>0$.

[TABLE]

In other words, if most non-zero coefficients of $\theta/\sigma$ are logarithmically small (at some power larger than $1/2$), it is still possible estimate the order of magnitude of $\|\theta\|_{0}$ up to some polylog multiplicative terms. 5. (e)

More generally, consider $\Delta=\|\theta\|_{0}(1-\tfrac{1}{u_{n}})$ with $u_{n}\to\infty$.

[TABLE]

For instance take $u_{n}=n^{\zeta}$ for $\zeta\in(0,\gamma)$. Even if most non-zero components of $\theta$, are polynomially small, it is still possible to distinguish $\theta$ from zero, but it is just possible to estimate $\log(\|\theta\|_{0})$ up to a multiplicative constant.

Finally, let us emphasize that all these convergence rates are optimal in the sense of Corollaries 3 and 4.

Comparison with the literature. In [8], Cai et al. consider an asymptotic framework where $\|\theta\|_{0}=n^{\gamma}$ with $\gamma\in(0,1/2)$ and $\theta$ only takes the values [math] and $\sigma\sqrt{2r\log(n)}$ for some $r>0$. These authors obtain convergence rates similar to Case (i) above but with explicit optimal constant $c(\alpha,\beta)$. In [7], Cai and Jin consider an asymptotic framework where the non zero components of $\theta$ are sampled according to a fixed distribution with a smooth density $h$ in the sense that its characteristic function decays at rate not slower than $t^{-\alpha}$ for some $\alpha>2$. Their estimator $\widetilde{k}$ [7, Sect. 3.1] achieves a relative convergence rate of order $\log^{-\alpha/2}(n)$. However, if $h$ does not satisfy an uniform smoothness assumption, then $\widetilde{k}$ can be inconsistent. According to Case (ii,b), when $h$ is continuous at [math], the relative convergence rate of our estimator $\widehat{k}$ is of order $\frac{\log\log(n)}{\sqrt{\log(n)}}$. This rate is slightly slower than that of Cai and Jin when $h$ is highly smooth, but our estimator is not tailored to vectors $\theta$ that are sampled according to a smooth distribution and is valid for all $\theta$. This difference in the optimal rates highlights that our problem is qualitatively not the same as theirs in relevant cases.

4 Sparsity testing with unknown variance

In this part, we consider the problem of testing the sparsity of $\theta$ when the noise level $\sigma$ is unknown. For the sake of simplicity, it is assumed that $\sigma$ belongs to some fixed interval $[\sigma_{-},\sigma_{+}]$ where $0<\sigma_{-}<\sigma_{+}$ are known. This assumption is not restrictive since, in most interesting settings, one may build a data-driven interval $[\widehat{\sigma}_{-},\widehat{\sigma}_{+}]$ containing $\sigma$ with large probability and such that the ratio $\widehat{\sigma}_{+}/\widehat{\sigma}_{-}$ remains bounded. See below for further explanations.

In this section and in the corresponding proofs, we denote $\mathbb{P}_{\theta,\sigma}$ the distribution of $Y$. Given two integers $k_{0}\geq 0$ and $\Delta>0$, we consider the sparsity testing problem with unknown variance

[TABLE]

Given a test $T$, let us define its risk $R_{\mathrm{var}}(T;k_{0},\Delta,\rho)$ for the problem (38) by

[TABLE]

and its $\gamma$-separation distance $\rho_{\gamma,\mathrm{var}}(T)$ by

[TABLE]

Finally, the minimax separation distance for the problem with unknown variance is defined by

[TABLE]

4.1 Detection problem ($k_{0}=0$)

Before turning to the general case, let us first restrict ourselves to the signal detection problem. To the best of our knowledge, the minimax separation distances for unknown variance have not been derived yet. Besides, this provides an introduction to the general case. Obviously, the problem with unknown variance is at least as difficult as the initial problem (4) so that, for all $\Delta$, $\rho^{*}_{\gamma,\mathrm{var}}[k_{0},\Delta]\geq\sigma_{+}\rho^{*}_{\gamma}[k_{0},\Delta]$. Our purpose is to pinpoint the range of $\Delta$ such that $\rho^{*}_{\gamma,\mathrm{var}}[k_{0},\Delta]$ is of order $\rho^{*}_{\gamma}[k_{0},\Delta]$ so that the the knowledge of the variance is not critical and the range of $\Delta$ such that $\rho^{*}_{\gamma,\mathrm{var}}[k_{0},\Delta]$ is much larger than $\rho^{*}_{\gamma}[k_{0},\Delta]$ so that the knowledge of the variance effectively makes the testing problem easier.

Proposition 4**.**

Fix any $\gamma<0.25$. There exists two positive constants $c_{\gamma}$ and $c^{\prime}_{\gamma}$ such that the following holds For any $\Delta\leq\sqrt{n}$, we have

[TABLE]

For any $\eta<1/3$ and any $\Delta\in[\sqrt{n},(\tfrac{1}{3}-\eta)n]$,

[TABLE]

where the constant $c_{\gamma,\eta}$ and $c^{\prime}_{\gamma,\eta}$ only depend on $\gamma$ and $\eta$.

For $\Delta\leq\sqrt{n}$, the minimax separation distance is the same as for known variance. This can be achieved, for instance, by a generalization of the Higher Criticism to the unknown variance setting as explained in Section 4.3.

For $\Delta$ between $\sqrt{n}$ and $n/3$, $\rho_{\gamma,\mathrm{var}}^{*2}[0,\Delta]$ is of order $\sqrt{\Delta n^{1/2}}$ which is much larger than the squared separation distance $\sqrt{n}$ for known variance. When $\sigma$ is known, a near optimal test amounts to reject the null hypothesis when $S_{2}=\|Y\|_{2}^{2}/\sigma^{2}-n$ is large in front of $\sqrt{n}$. Under the null, $S_{2}+n$ follows a $\chi^{2}$ distribution with $n$ degrees of freedom whereas, under the alternative, $S_{2}+n$ follows a non-central $\chi^{2}$ distribution with non-centrality parameter $\|\theta\|_{2}^{2}/\sigma^{2}$ so that the test is powerful when $\|\theta\|_{2}^{2}$ is large in front of $\sigma^{2}\sqrt{n}$. When $\sigma$ is unknown, one cannot simply rely on the second moment of $Y$ and higher order moments are needed. For instance, a test achieving the separation distance (43) is based on the statistic

[TABLE]

Under the null, it follows from Chebychev inequality that $S_{4}=O_{P}(n^{-1/2})$. Under the alternative, $\operatorname{\mathbb{E}}_{\theta,\sigma}[\|Y\|_{2}^{2}]=\|\theta\|_{2}^{2}+n\sigma^{2}$ and $\operatorname{\mathbb{E}}_{\theta,\sigma}[\|Y\|_{4}^{4}]=\|\theta\|_{4}^{4}+6\sigma^{2}\|\theta\|_{2}^{2}+3n\sigma^{2}$ so that, one may expect that $S_{4}$ is of order

[TABLE]

by Cauchy-Schwarz inequality. As a consequence, one may expect that $S_{4}$ takes significantly larger values when $(n-3\|\theta\|_{0})\|\theta\|_{4}^{4}$ is large in front of $\sqrt{n}$. When $n-3\Delta$ is of order $n$, this occurs when $\|\theta\|_{2}^{2}$ is larger than $\sqrt{\Delta n^{1/2}}$. See the proof of Proposition 4 for further details.

Conversely, the proof of the minimax lower bound (43) also proceeds from moments arguments. For known variance $\sigma=1$, one builds a prior probability measure $\nu$ on $\theta$ supported by $\mathbb{B}_{0}[\Delta]$ such that the expectation of $\sum_{i=1}^{n}Y_{i}$ is the same under $\int\operatorname{\mathbb{P}}_{\theta,\sigma}\nu(d\theta)$ and $\operatorname{\mathbb{P}}_{0,\sigma}$. When the variance is unknown, one may choose $\sigma_{1}\neq\sigma_{0}$ such that all expectations $\sum_{i=1}^{n}Y^{q}_{i}$ for $q=1,2,3$ are matching under $\int\operatorname{\mathbb{P}}_{\theta,\sigma_{1}}\nu(d\theta)$ and $\operatorname{\mathbb{P}}_{0,\sigma_{0}}$. As explained in the proof of Theorem 3, these moment matching properties translate into a smaller total variation between $\int\operatorname{\mathbb{P}}_{\theta,\sigma_{1}}\nu(d\theta)$ and $\operatorname{\mathbb{P}}_{0,\sigma_{0}}$ which in turn implies that the separation distance $\rho^{*}_{\gamma,\mathrm{var}}[0,\Delta]$ is large.

Proposition 4 above characterizes the signal detection separation distance for all $\Delta$ small in front of $n/3$. For $\Delta=cn$ with $c<1/3$, $\rho_{\gamma,\mathrm{var}}^{*2}[0,\Delta]$ is of order $n^{3/4}$. One may then wonder if $\rho_{\gamma,\mathrm{var}}^{*2}[0,\Delta]$ remains of order $n^{3/4}$ for all $\Delta\in(n/3,n]$. This turns out to be false. In fact, $\rho_{\gamma,\mathrm{var}}^{*2}[0,n]$ is of order $(\sigma^{2}_{+}-\sigma^{2}_{-})n$. Indeed, let $\nu$ denote the centered normal distribution with variance $(\sigma^{2}_{+}-\sigma_{-}^{2})I_{n}$. When $\theta$ is sampled according to $\nu$ and for $\sigma=\sigma_{-}$, the marginal distribution of $Y$ is $\mathbb{P}_{0,\sigma^{+}}$. As a consequence, it is impossible to distinguish $\theta=0$ from $\theta\sim\nu$ for which $\|\theta\|_{2}^{2}$ is of order $(\sigma^{2}_{+}-\sigma^{2}_{-})n$. This entails that $\rho_{\gamma,\mathrm{var}}^{*2}[0,n]$ is at least of order $(\sigma^{2}_{+}-\sigma^{2}_{-})n$.

In fact, the squared minimax separation distance $\rho_{\gamma,\mathrm{var}}^{*2}[0,\Delta]$ jumps above $n^{3/4}$ well before $\Delta=n$ as stated by the next proposition.

Proposition 5**.**

Consider any $0\leq\gamma\leq 0.25$. Fix any $\eta>0$ arbitrarily small and take $\Delta=\lfloor(\frac{1}{3}+\eta)n\rfloor$. For $n$ large enough, we have

[TABLE]

for some constant $c_{\eta}>0$ only depending on $\eta$.

As a consequence, the detection problem become much more difficult when $\Delta$ is above $n/3$ and the condition on $\Delta$ in Proposition 4 is tight. In comparison to the proof of the lower bound (43), for $\Delta$ larger than $n/3$, it is possible to define a prior measure $\nu$ supported on $\mathbb{B}_{0}[\Delta]$, $\sigma_{0}$ and $\sigma_{1}$ such that all expectations $\sum_{i=1}^{n}Y^{q}_{i}$ for $q=1,\ldots,5$ are matching under $\int\operatorname{\mathbb{P}}_{\theta,\sigma_{1}}\nu(d\theta)$ and $\operatorname{\mathbb{P}}_{0,\sigma_{0}}$. Matching these five moments then allows to recover the $n^{5/6}$ rate. See the proof of Proposition 5 for details.

To summarize, for $\Delta\leq\sqrt{n}$ the minimax detection distance is the same as for known variance. For $\Delta\in[\sqrt{n},cn]$ with $c<1/3$ the square minimax detection distance is of order $\sqrt{\Delta n^{1/2}}$ which is larger than its counterpart for known variance. For $\Delta>cn$ with $c>1/3$, the difficulty of the testing problem greatly increases.

In view of this phenomenon, we shall restrict ourselves, for the general sparsity testing problems, to values $(k_{0},\Delta)$ such that $k_{0}+\Delta\leq cn$ where $c$ is some constant small enough.

4.2 Lower bounds

For $\Delta\leq\sqrt{n}\vee k_{0}$ we simply use the lower bound $\rho_{\gamma,\mathrm{var}}^{*2}[k_{0},\Delta]\geq\rho_{\gamma}^{*2}[k_{0},\Delta]$ (where $\rho_{\gamma}^{*2}[k_{0},\Delta]$ is defined for known $\sigma=\sigma_{+}$). The following corollary is then a direct consequence of Theorem 1.

Corollary 5**.**

Consider any $\gamma\leq 0.5$. For any $k_{0}\leq\sqrt{n}$ and $\Delta\leq n-k_{0}$, we have

[TABLE]

There exists a numerical constant $c>0$ such that the following holds. For any $k_{0}>\sqrt{n}$ and $\Delta\leq k_{0}\wedge(n-k_{0})$, we have

[TABLE]

Additional work is needed to pinpoint the minimax separation distance $\rho_{\gamma,\mathrm{var}}^{*}[k_{0},\Delta]$ for $\Delta\geq\sqrt{n}\vee k_{0}$. As for known variance, there are two different regimes depending whether $k_{0}\leq\sqrt{n}$ or $k_{0}>\sqrt{n}$.

Theorem 3**.**

Consider any $0\leq\gamma\leq 0.25$. For any $0\leq k_{0}\leq\sqrt{n}$ and $\max(\sqrt{n},48)\leq\Delta\leq n-k_{0}$, we have

[TABLE]

where $c$ is a numerical constant.

For $k_{0}\leq\sqrt{n}$ and $\Delta\geq\sqrt{n}$, the separation distance $\rho_{\gamma,\mathrm{var}}^{*2}[k_{0},\Delta]$ is the same as in the signal detection setting $\rho_{\gamma,\mathrm{var}}^{*2}[0,\Delta]$. In comparison to $\rho_{\gamma}^{*2}[k_{0},\Delta]$, the squared distance $\sqrt{n}$ has increased up to $\sqrt{\Delta n^{1/2}}$. The intuition behind Theorem 3 has been already described below Proposition 4.

Theorem 4**.**

There exist three positive constants $c_{1}$, $c_{2}$, and $c_{3}$ such that the following holds. Assume that $n/c_{1}\geq\Delta\geq c_{1}k_{0}\geq c_{1}\sqrt{n}$ and that $n\geq c_{2}$. Then, we have

[TABLE]

In the known variance setting, the squared separation distance is of order $\frac{k_{0}}{\log(1+k_{0}/\sqrt{n})}$. The price to pay for not knowing the variance is a multiplicative factor of order $\sqrt{\Delta/k_{0}}$.

Contrary to the proof of Theorem 1 for known variance, it is difficult to follow here a moment matching approach. Given two suitable prior distributions $\mu_{0}^{\otimes n}$ and $\mu_{1}^{\otimes n}$ on $\theta$ and variances $\sigma_{0}^{2}$ and $\sigma_{1}^{2}$ in such a way that $\mu_{0}^{\otimes n}$ is almost supported in $\mathbb{B}_{0}[k_{0}]$ and $\mu_{1}^{\otimes n}$ is almost supported in $\mathbb{B}_{0}[k_{0}+\Delta,k_{0},\rho]$, the goal is to prove that the two marginal distribution of $Y$, $\int\operatorname{\mathbb{P}}_{\theta,\sigma_{0}}\mu^{\otimes n}_{0}(d\theta)$ and $\int\operatorname{\mathbb{P}}_{\theta,\sigma_{1}}\mu^{\otimes n}_{1}(d\theta)$ are close to each other in total variation distance. Since the two last measures are product measures, this is equivalent to proving that the densities $\pi_{0}(x):=\int\phi(\frac{t-x}{\sigma_{0}})\mu_{0}(dx)$ and $\pi_{1}(x):=\int\phi(\frac{t-x}{\sigma_{1}})\mu_{1}(dx)$ are close in $l_{1}$ distance (recall that $\phi(.)$ denotes the density of the standard normal distribution). It is difficult to obtain an analytic expression of the $l_{1}$ distance between two mixture distribution and hence one cannot directly choose the measure $\mu_{0}$ and $\mu_{1}$ minimizing this $l_{1}$ distance. As performed earlier in e.g.  [10, 29], we choose instead $\mu_{0}$ and $\mu_{1}$ in such a way that the Fourier transforms $\widehat{\pi}_{0}$ and $\widehat{\pi}_{1}$ are matching for all frequencies small enough. Afterwards, we prove that this particular choice of $\mu_{0}$ and $\mu_{1}$ makes the $l_{1}$ distance between $\pi_{0}$ and $\pi_{1}$ small. Although the general approach is not new, the control of the $l_{1}$ distance is more delicate than in previous work, especially in the regime where $k_{0}$ is close to $\sqrt{n}$. In the proof, our implicit construction of the prior distributions $\mu_{0}$ may be of independent interest.

4.3 Upper bounds

In this subsection, we build matching upper bounds for all $(k_{0},\Delta)$ such that $k_{0}+\Delta\leq cn$ where $c$ a numerical constant small enough. Indeed, when $\Delta$ is of order $n$, it has been proved in Proposition 5 that the detection problem becomes much more difficult, so that there is no hope to find tests matching Theorem 3 and Theorem 4 when $k_{0}+\Delta$ is too large. Note that, in the regime $k_{0}+\Delta\leq cn$, one may construct a data-driven confidence interval of $\sigma$ so that the knowledge of the fixed interval $[\sigma_{+},\sigma_{-}]$ is not really critical. In Appendix A, we provide such a confidence interval and we briefly explain how to how to extend the testing procedures to completely unknown variances $\sigma\in\mathbb{R}^{+}$.

Throughout this subsection, we consider some fixed $\alpha$ and $\beta$ in $(0,1)$.

4.3.1 Adaptive Higher Criticism Statistic

The principle underlying the Higher Criticism is to compare the number $N_{t}$ of components of $Y$ larger than $t$ in absolute value to an upper bound of their expectation under the null, namely $k_{0}+(n-k_{0})\Phi(t/\sigma)$. This is why we adapt this test by plugging a suitable estimator of $\sigma$ and adding some correcting terms accounting for the variance estimation error. Let

[TABLE]

where we recall that $\overline{\varphi}_{n}$ is the empirical characteristic function (16) of $Y$. Let us briefly explain the idea behind this definition by replacing $\overline{\varphi}_{n}(v)$ by its expectation $\overline{\varphi}(v)$ (16). Intuitively, $\widehat{\sigma}^{2}$ is expected to be of order

[TABLE]

so that when $\tfrac{1}{n}\sum_{i}\cos(v\theta_{i})$ is close to one, $\widehat{\sigma}^{2}$ should be close to $\sigma^{2}$. Estimation of $\sigma$ based on the empirical characteristic function has been first tackled by Cai and Jin [27, 7]. Nevertheless, our estimator (47) differs from theirs, as we do not assume that the non-zero components of $\theta$ are sampled from a smooth distribution.

Defining $t_{*,\alpha}^{HC,\mathrm{var}}:=\lceil 2\sqrt{2\log(\frac{4n}{\alpha})}\rceil$, we consider the test $T^{HC,\mathrm{var}}_{\alpha,k_{0}}$ that rejects the null hypothesis, if either $N_{\sigma_{+}t_{*,\alpha}^{HC,\mathrm{var}}}\geq k_{0}+1$ or if for some integer $t\geq 1$,

[TABLE]

where

[TABLE]

In comparison to the original calibration parameter $u_{t,\alpha}^{HC}$, the third term is new and accounts for the estimation error of $\sigma^{2}$.

Theorem 5**.**

Let $C$ be any constant larger than $1$. There exist constants $c$, $c^{\prime}_{\alpha}$, $c^{\prime\prime}_{\beta,\sigma_{+}/\sigma_{-},C}$, and $c^{\prime\prime\prime}_{\alpha,\beta}$ such that the following holds. If $n\geq c^{\prime}_{\alpha}$ and $k_{0}\leq n/c$, the type I error probability of $T^{B,\mathrm{var}}_{\alpha,k_{0}}$ is smaller than $\alpha$, that is

[TABLE]

Now assume that $n\geq c^{\prime\prime}_{\beta,\sigma_{+}/\sigma_{-},C}$. Any $\theta\in\mathbb{R}^{n}$ satisfying $\|\theta\|_{0}\leq n/c$,

[TABLE]

for some $q\in[1,n-k_{0}]$ and

[TABLE]

belongs to the high probability rejection region of $T^{B,\mathrm{var}}_{\alpha,k_{0}}$, that is $\operatorname{\mathbb{P}}_{\theta,\sigma}[T^{HC,\mathrm{var}}_{\alpha,k_{0}}=0]\leq\beta$.

Condition (52) aside, the behavior of $T^{HC,\mathrm{var}}_{\alpha,k_{0}}$ is similar to the one of $T^{HC}_{\alpha,k_{0}}$ as stated in Proposition 7. In fact, Condition (52) allows to bound the term $\frac{1}{n}\sum_{i}\cos(v\theta_{i})$ in (48) and ensures that $|\widehat{\sigma}^{2}-\sigma^{2}|$ is, with high probability, at most of order $\tfrac{k_{0}}{n\log(1+k_{0}/\sqrt{n})}$. When this condition (52) is not met, we are unable to control the behavior of the adaptive Higher Criticism test. Nevertheless, it turns out that parameters $\theta$ not satisfying (52) belong to the high-probability rejection region of the test $T^{B,\mathrm{var}}_{\alpha,k_{0}}$ described below so that a combination of $T^{HC,\mathrm{var}}_{\alpha,k_{0}}$ and $T^{B,\mathrm{var}}_{\alpha,k_{0}}$ achieves similar performances to the original Higher Criticism test $T^{HC,\mathrm{var}}_{\alpha,k_{0}}$. At the end of the section, the constant $C$ in Theorem 5 will be carefully chosen to put the three tests $T^{HC,\mathrm{var}}$, $T^{B,\mathrm{var}}$ and $T^{I,\mathrm{var}}$ together.

4.3.2 Detecting the signal in the bulk distribution

Analogously to the above extension of the Higher-Criticism test, it would be natural to plug a variance estimator $\widehat{\sigma}^{2}$ in the statistic $Z(s)$ (15) and then to build a test based on this data-driven statistic. Unfortunately, it turns out that the estimation error for such $\widehat{\sigma}$ is not negligible in our setting. Such a phenomenon is not unexpected as we have proved in Theorem 4 that no test in the unknown variance setting can perform as well as $T_{\alpha,k_{0}}^{B}$ for known $\sigma$.

This is why we define a new statistic which is almost invariant with respect to the noise variance. Denoting $P_{B}$ the linear polynom $P_{B}(\xi):=4\xi-3$, we define, for $s>0$, the statistic $Z^{\mathrm{var}}(s)$

[TABLE]

The polynom $P_{B}$ has been defined in such a way that $\int_{0}^{1}P_{B}(\xi)\xi^{2}d\xi=0$. To understand the rationale behind $Z^{\mathrm{var}}(s)$, let us assume that $\overline{\varphi}_{n}(s\xi)$ is close to its expectation $\overline{\varphi}(s\xi)$. Since for $x$ close to $1$, $\log(x)$ is approximately $x-1$, we obtain

[TABLE]

where g(x)=\int_{0}^{1}P_{B}(\xi)\big{(}\cos(\xi x)-1\big{)}d\xi. For small $x$, a Taylor expansion of the $\cos$ function enforces that $g(x)\approx\int_{0}^{1}P_{B}(\xi)[-\xi^{2}\tfrac{x^{2}}{2}+\xi^{4}\tfrac{x^{4}}{12}]d\xi=x^{4}\int_{0}^{1}P_{B}(\xi)\tfrac{\xi^{4}}{12}d\xi>0$. For larger $x$ (in absolute value), one can prove that $g(x)$ is positive and bounded away from zero. As a consequence, $\sum_{i=1}^{n}g(s\theta_{i}/\sigma_{+})$ behaves like $\sum_{i=1}^{n}(s\theta_{i}/\sigma_{+})^{4}\wedge 1$ and approximates $\|\theta\|_{0}$. This informal discussion is made rigorous in the proof of Theorem 6 below. In practice, we set

[TABLE]

and we define $T^{B,\mathrm{var}}_{\alpha,k_{0}}$ as the test rejecting the null hypothesis for large values of $Z^{\mathrm{var}}(s_{k_{0}}^{\mathrm{var}})$, that is when

[TABLE]

Theorem 6**.**

There exist numerical constants $c$, $c^{\prime}$, and $c^{\prime\prime}_{\alpha,\beta}$ such that the following holds. Assume that $n\geq c$ and that $k_{0}\leq c^{\prime}n$. For any $k_{0}$-sparse vector $\theta$, the type I error probability of $T^{B,\mathrm{var}}_{\alpha,k_{0}}$ is small, that is

[TABLE]

Any $\theta\in\mathbb{R}^{n}$ such that $\|\theta\|_{0}\leq c^{\prime}n$, and

[TABLE]

belongs to the high probability rejection region of $T^{B,\mathrm{var}}_{\alpha,k_{0}}$, that is

[TABLE]

The sufficient condition (57) for $T^{B,\mathrm{var}}_{\alpha,k_{0}}=1$ to be powerful corresponds to the heuristics described above. This condition will be the main ingredients towards matching the $\sigma_{+}^{2}\tfrac{\sqrt{\Delta k_{0}}}{\log(1+k_{0}/\sqrt{n})}$ separation distance of Theorem 4.

The main downside to the above theorem is the presence of the small term $\|\theta\|_{1}/(\sigma_{+}n^{4})$ in the type I and type II error probabilities. Although for typical parameters $\theta$ this term will be negligible, this makes the supremum of the type I error bound (56) over all $\theta\in\mathbb{B}_{0}[k_{0}]$. In Section 4.3.4, we sketch a trimming approach which amounts to first discard components large components $Y$ and then apply the test to the trimmed vector $\tilde{Y}$. The $l_{1}$ norm of the corresponding trimmed parameter $\tilde{\theta}$ is then small enough so that the type I and type II error probabilities are uniformly controlled.

4.3.3 Intermediary regimes

As for $T^{B}_{\alpha,k_{0}}$, one cannot easily adapt $T^{I}_{\alpha,k_{0}}$ by plugging an estimator of $\sigma$. Following the same approach as above we modify the statistic by considering the logarithm of the empirical characteristic function and multiplying it by some suitable polynom.

As the following test aims at discovering intermediary signals whose signature is neither in the bulk of the empirical distribution of $(Y_{i})$ nor in its extreme values, we restrict ourselves to the case $k_{0}\geq 20\sqrt{n}$ (as for $T^{I}_{\alpha,k_{0}}$). Consider the dyadic collection $\mathcal{L}_{k_{0}}$ defined in Section 2.2.3. For $l\in\mathcal{L}_{k_{0}}$, let

[TABLE]

Note that, if $w_{l}$ is defined as in (21) for $T^{I}_{\alpha,k_{0}}$, the definition of $r_{k_{0},l}$ is slightly different. Equipped with this notation, we consider the statistic

[TABLE]

where P_{l}(t)=\gamma_{l}\big{[}\zeta_{l}t^{2}-\kappa_{l}\big{]} with

[TABLE]

The purpose of this polynom $P_{l}$ is to cancel the term $\int_{-1}^{1}P_{l}(r_{k_{0},l}\xi)\phi(r_{k_{0},l}\xi)\xi^{2}d\xi$. Heuristically, $\log[\overline{\varphi}_{n}(w_{l}\xi/\sigma_{+})_{+}]$ should be close to

[TABLE]

Since $P_{l}(r_{k_{0},l}\xi)\phi(r_{k_{0},l}\xi)$ is orthogonal to $\xi^{2}$, we expect that

[TABLE]

Each term of this sum is zero for $\theta_{i}=0$. More generally, we show in the proof of Theorem 7 that, when $\theta$ does not contain too many large coefficients, this sum approximates the number of coefficient larger than $r_{k_{0},l}^{2}/w_{l}$.

Finally, let $T^{I,\mathrm{var}}_{\alpha,k_{0}}$ be the test rejecting the null hypothesis, if for some $l\in\mathcal{L}_{k_{0}}$, $V^{\mathrm{var}}(r_{k_{0},l},w_{l})$ is large enough, that is

[TABLE]

Theorem 7**.**

There exist numerical constants $c$, $c^{\prime}$, $c^{\prime\prime}_{\alpha,\beta}$, and $c^{\prime\prime\prime}_{\alpha,\beta}$ such that, for any $C>2$, the following holds. Assume that $n\geq c$ and that $k_{0}\leq c^{\prime}n$. For any $k_{0}$-sparse vector $\theta$, the type I error probability of $T^{I,\mathrm{var}}_{\alpha,k_{0}}$ is small, that is

[TABLE]

Recall $s_{k_{0}}^{\mathrm{var}}$ defined in (54). Any parameter $\theta\in\mathbb{R}^{n}$ satisfying $\|\theta\|_{0}\leq c^{\prime}n$ and the two following properties

[TABLE]

belongs to the high probability rejection region of $T^{I,\mathrm{var}}_{\alpha,k_{0}}$, that is

[TABLE]

Condition (63) for $T^{I,\mathrm{var}}_{\alpha,k_{0}}$ to be powerful is analogous to Condition (104) for $T^{I}_{\alpha,k_{0}}$ in the known variance setting except that $q$ is now restricted to be larger than $k_{0}^{2}/n$. This restriction will turn out to be benign except when $k_{0}$ is too close to $n$. Also, contrary to Proposition 9, $\theta$ is assumed to contain less than $Ck_{0}$ coefficients larger than $\sigma_{+}/s_{k_{0}}^{\mathrm{var}}$ (which is of order $\sigma_{+}\log(k_{0}/\sqrt{n})^{-1/2}$). Again, this restriction is not a serious issue as $T^{B,\mathrm{var}}_{\alpha,k_{0}}$ is powerful for such $\theta$ not satisfying this assumption.

4.3.4 Combination of the tests

For any integers $k_{0}\geq 0$ and $q>0$, define $\psi_{k_{0},q}^{\mathrm{var}}>0$ by

[TABLE]

Let $T^{C,\mathrm{var}}_{\alpha,k_{0}}$ denote the aggregation of the three previous tests, that is

[TABLE]

and

[TABLE]

As pointed out above, it is not possible to control uniformly the type I error probability of this test as such probabilities depend on the $l_{1}$ norm of $\theta$. This is why introduce a trimmed version of this test by removing large components of $Y$. Given $z>0$ and $V\in\mathbb{R}^{n}$, let $\mathcal{S}(z;V)=\{i\in[n],\ |V_{i}|>(z+1)\sigma_{+}n^{2}\}$. Let $U\sim\mathcal{U}[0,1]$ be an uniformly distributed random variable independent of $Y$. We write $\mathcal{S}(U,Y)=\mathcal{S}[(U+1)\sigma_{+}n^{2};Y]$ for the coordinates $i$ such that $|Y_{i}|>(U+1)\sigma_{+}n^{2}$. Let $\widetilde{Y}(\mathcal{S}(U,Y)):=(Y_{i}),i\in([n]\setminus\mathcal{S}(U,Y))$ be the sub vector of $Y$ of size $n-|\mathcal{S}(U,Y)|$. Finally, we define the trimmed test $\overline{T}^{C,\mathrm{var}}_{\alpha,k_{0}}$ rejecting the null hypothesis if either $k_{0}-|\mathcal{S}(U,Y)|$ is negative or if the test $T^{C,\mathrm{var}}_{\alpha,k_{0}-|\mathcal{S}(U,Y)|}$ applied to the size $n-|\mathcal{S}(U,Y)|$ vector $\widetilde{Y}(\mathcal{S}(U,Y))$ rejects the null hypothesis.

We use a random threshold $(U+1)\sigma_{+}n^{2}$ instead of a deterministic one to make the subset $\mathcal{S}$ of trimmed variable almost independent from $Y$, which facilitate the analysis of the two-step procedure $\overline{T}^{C,\mathrm{var}}_{\alpha,k_{0}}$.

Corollary 6**.**

Fix any $\xi\in(0,1)$. There exist positive constants $c$, $c^{\prime}$, $c^{\prime\prime}_{\alpha,\beta,\xi}$ and $c^{\prime\prime\prime}_{\alpha,\beta,\xi}$ such that the following holds. Consider any $k_{0}\leq n^{1-\xi}$ and $n\geq c$. Then, for any $\theta\in\mathbb{B}_{0}[k_{0}]$, one has

[TABLE]

Moreover, $\operatorname{\mathbb{P}}_{\theta,\sigma}[\overline{T}^{C,\mathrm{var}}_{\alpha,k_{0}}=1]\geq 1-\beta-\frac{c^{\prime}\log(n)}{n}$ for any vector $\theta$ satisfying $\|\theta\|_{0}\leq c^{\prime}n$ and

[TABLE]

Also, $\operatorname{\mathbb{P}}_{\theta,\sigma}[\overline{T}^{C,\mathrm{var}}_{\alpha,k_{0}}=1]\geq 1-\beta-\frac{c^{\prime}\log(n)}{n}$ for any vector $\theta$ satisfying

[TABLE]

As a consequence, for $k_{0}\leq n^{1-\xi}$ (and $\xi$ is an arbitrary constant in $(0,1)$), $\overline{T}^{C}_{\alpha,k_{0}}$ simultaneously achieves the minimax separation distance for all $\Delta$ such that $k_{0}+\Delta\leq cn$ where $c$ is constant small enough.

Building on the statistics introduced in this section, one can then construct an adaptive estimator of the sparsity for unknown variance in the spirit of what has been done in Section 3. For reasons of space, we do not pursue in this direction.

5 Discussion

5.1 Other noise distributions

Some of our testing procedures heavily rely on the assumption that the noise’s distribution is Gaussian. For instance, the behavior of the Bulk and intermediary statistics depends on the exact form of the characteristic function of the noise. The radical change in the rates between the known variance case, and the unknown variance case, is already eloquent enough on the importance of knowing the exact shape of the noise distribution - even a slight deformation of the noise distribution by changing the variance has a strong effect on the minimax separation distances. We may consider two different extensions to non-Gaussian noises:

The noise distribution is not Gaussian but is explicitly known. For the sake of discussion, let us also assume that it is symmetric. In that case, one could adapt the higher criticism statistic by replacing $\Phi(.)$ by the survival function of this distribution. Also, both the bulk and intermediary statistic could be accommodated by replacing $\exp(-\xi^{2}w^{2}/2)$ in (14) by the characteristic function of the noise distribution. Nevertheless, some additional work would be needed to adapt the lower bounds 2. 2.

Only an upper bound of the tail distribution of the noise is known. For instance, the noise is only assumed to be sub-Gaussian with a bounded sub-Gaussian norm. In that situation, one cannot rely anymore on its characteristic function. Nevertheless, one could adapt some signal detection tests [3] to build “infimum test” [19, 37] such as those described in the introduction. From rough calculations, it seems that the corresponding test would achieve the optimal separation distances up to polylogarithmic multiplicative terms. It remains an open problem to understand whether this polylog loss is intrinsic or not.

5.2 Other models

The same general roadmap can be pursued to estimate discrete functionals in many other problems, including rank estimation in matrix regression and matrix completion models, smoothness estimation in the density framework, number of clusters estimation in model-based clustering,…. A prominent example is sparsity estimation in the high-dimensional linear regression model. Let $Y\in\mathbb{R}^{n}$, ${\bf X}\in\mathbb{R}^{n\times p}$ be such that

[TABLE]

where the parameter $\theta\in\mathbb{R}^{p}$ is unknown and $\epsilon=(\epsilon_{i})$ is made of centered independent normal distributions with variance $\sigma^{2}$. In the specific case where $n=p$ and ${\bf X}$ is the identity matrix, it is is equivalent to Gaussian vector model (1). Estimation of $\theta$ under sparsity assumptions has received a lot of attention in the last decade [6]. In the specific case where the entries of $\mathbf{X}$ are independently sampled according to the standard normal distribution, the minimax separation distances for the detection problem has been derived in [24, 2]. For the purpose of building adaptive confidence intervals, Nickl and van de Geer [37] have introduced and analyzed sparsity testing procedures. However, the optimal separation distances for the sparsity testing problem remain unknown (except in some specific regimes). Further work is therefore needed to establish the minimax separation distances and to construct adaptive sparsity tests and sparsity estimators.

Acknowledgements.

The work of A. Carpentier is supported by the Deutsche Forschungsgemeinschaft (DFG) Emmy Noether grant MuSyAD (CA 1488/1-1). The authors thank Christophe Giraud for careful rereading and insightful suggestions on the presentation of the results.

Appendix A Estimation of $\sigma_{-}$ and $\sigma_{+}$ and full adaptation to unknown $\sigma$

The purpose of this section is to exhibit a confidence interval of $\sigma$ that. This allows us to first estimate $[\hat{\sigma}_{-},\hat{\sigma}_{+}]$ and plug this confidence interval in the testing procedures of Section 4.

Lemma 1**.**

There exists some universal constant $c>0$ such that the following holds for any $\theta\in\mathbb{B}_{0}[n/2]$. Define

[TABLE]

$\hat{\sigma}_{+}:=2.2\tilde{\sigma}~{}~{}\text{and}~{}~{}\hat{\sigma}_{-}:=\tilde{\sigma}/16.$* With probability higher than $1-2e^{-cn}$, we know that*

[TABLE]

Outside an event of exponentially small probability, $[\hat{\sigma}_{-},\hat{\sigma}_{+}]$ only takes seven possible values. Then, conditioning on each of these seven events, one analyzes the behavior of the tests $T_{\alpha,k_{0}}^{HC,\mathrm{var}}$, $T_{\alpha,k_{0}}^{B,\mathrm{var}}$, and $T_{\alpha,k_{0}}^{I,\mathrm{var}}$ to control the risk of the corresponding fully data-driven procedures.

Proof of Lemma 1.

The proof follows closely that of Proposition 1 in [15]. For the sake of simplicity, we assume that $n$ is even. Let $\mathcal{S}$ be a set of size $n/2$ that does not intersect with the support of $\theta$. Then,

[TABLE]

the last random variable following a $\chi^{2}$ distribution with $n$ degrees of freedom. By [5], we know that

[TABLE]

where $c$ is some positive universal constant. Next, let $\mathcal{G}$ the collection of subsets of $[n]$ of size $n/2$. We shall control the deviations of the random variables $Z_{G}:=\frac{1}{\sigma^{2}}\sum_{i\in G}Y_{i}^{2}$ uniformly over all $G\in\mathcal{G}$. Fix any $G\in\mathcal{G}$. The random variable $Z_{G}$ follows a $\chi^{2}$ distribution with $n/2$ degrees of freedom and non-centrality parameter $\sum_{i\in G}\theta_{i}^{2}/\sigma^{2}$. In particular, this distribution is stochastically larger than a (central) $\chi^{2}$ distribution with $n/2$ degrees of freedom. Let $Z$ be a random variable sampled according to this distribution. By Lemma 11.1 in [41], we know that for any $x>0$,

[TABLE]

Take $x=\binom{n}{n/2}^{-1}e^{-n/8}$. It follows that $\log(1/x)\leq n(\tfrac{1}{8}+\log(2))$. Taking an union bound over all $Z_{G}$ for $G\in\mathcal{G}$, we conclude that

[TABLE]

Since $\bar{\sigma}^{2}=\frac{2}{n}\sigma^{2}\inf_{G}Z_{G}$, this implies that, with high probability, $\bar{\sigma}^{2}\geq\frac{\sigma^{2}}{16e^{3/2}}$. We have proved that with high probability,

[TABLE]

The remainder of the proof follows easily. ∎

Appendix B Proofs of the results with known variance

In all the proofs in this section, we assume by homogeneity and without loss of generality that $\sigma=1$.

B.1 Proofs of the testing lower bounds with known variance

The minimax separation distance $\rho^{*}_{\gamma}[k_{0},\Delta]$ depends on $\gamma$, $n$, $k_{0}$ and $\Delta$. In these proofs, we shall relate the minimax separation distances for different values of the sample size. To make the arguments clearer, we explicit the dependency of it on the sample size and write $\rho^{*}_{\gamma}[n,k_{0},\Delta]$ instead of $\rho^{*}_{\gamma}[k_{0},\Delta]$ in this subsection.

Step 1 : Reduction of the problem.

We start by simple reduction arguments to narrow the range of parameters.

Lemma 2**.**

For any $k^{\prime}_{0}\leq k_{0}$,

[TABLE]

For any $\Delta^{\prime}\leq\Delta\leq n-k_{0}$,

[TABLE]

Finally,

[TABLE]

Proof of Lemma 2.

The second bound is a consequence of the inclusion $\mathbb{B}_{0}[k_{0}+\Delta,k_{0},\rho]\subset\mathbb{B}_{0}[k_{0}+\Delta^{\prime},k_{0},\rho]$. The third bound is also trivial. Let us turn to (67), consider any $\zeta>0$ arbitrarily small and let $r:=\rho_{\gamma}^{*}[n,k_{0},\Delta]+\zeta$. There exists a test $T$ satisfying $R[T;k_{0},\Delta,r]\leq\gamma$. For any $n-k_{0}+k^{\prime}_{0}$-dimensional vector $Y$ with mean $\theta$, extend it to $\tilde{Y}$ by adding $k_{0}-k^{\prime}_{0}$ components following independent standard normal distribution with mean $r$. Since $R[T;k_{0},\Delta,r]\leq\gamma$, we have

[TABLE]

implying that $\rho_{\gamma}^{*}[n-k_{0}+k^{\prime}_{0},k^{\prime}_{0},\Delta]\leq r$. Considering the infimum over all $\zeta>0$, we obtain (67). ∎

As a consequence of the above lemma, we obtain the following reduction.

Proposition 6**.**

Theorem 1 is true as soon as

[TABLE]

for any $k_{0}>\sqrt{n}$ and $32\sqrt{(n-k_{0})\wedge k_{0}}\leq\Delta\leq k_{0}$.

The proof of this reduction is postponed to the end of the subsection. In the sequel, we focus on (70–72). The first bound (70) has already been shown in [3]. For the sake of completeness, we shall provide a proof of it together with (71). Prior to this, we focus on (72).

Step 2. Le Cam’s method.

In this step, we explain the general strategy for proving the minimax lower bound, allowing us to introduce the main notation. We start by introducing probability measures on the space of parameters $\theta$. Fix some $\gamma\in(0,1/2)$. Define $\overline{k}_{0}=k_{0}-\Delta/2$, $k_{1}=k_{0}+\Delta$ (the sparsity of the alternative) and $\overline{k}_{1}=k_{0}+\Delta/2$.

Let $m\geq 1$, $M>0$, and $a_{m}>0$ be quantities whose values will be fixed later. Below, we shall build two symmetric probability measures $\mu_{0}$ and $\mu_{1}$ whose support is included in

[TABLE]

Given $\mu_{0}$ and $\mu_{1}$, consider the probability measures $\overline{\mu}_{0}$ and $\overline{\mu}_{1}$

[TABLE]

Let $\overline{\mu}_{0}^{\otimes n}$ and $\overline{\mu}_{1}^{\otimes}$ be the corresponding $n$-dimensional product measure. Note that, when $\theta\sim\overline{\mu}_{0}^{\otimes n}$, its number of non-zero coefficients follows a Binomial distribution with parameters $n$ and $\overline{k}_{0}$.

Finally, we define

[TABLE]

the marginal probability distribution of $Y$ when $\theta\sim\overline{\mu}^{\otimes n}_{0}$ (resp. $\theta\sim\overline{\mu}^{\otimes n}_{1}$). By Chebychev inequality,

[TABLE]

Similarly,

[TABLE]

With $\overline{\mu}_{1}^{\otimes n}$-probability larger than $1-9/32$, $\theta$ is therefore $k_{1}$-sparse and $d_{2}^{2}(\theta,\mathbb{B}_{0}(k_{0}))\geq\Delta a_{m}^{2}M^{2}/4$. Given any test $T$, we apply Fubini identity to lower bound its risk (5) as

[TABLE]

As a consequence, the minimax separation distance $\rho^{*}_{\gamma}[n,k_{0},k_{1}]$ is larger than $\Delta^{1/2}a_{m}M/2$, as soon as

[TABLE]

where $\delta:=0.55-\gamma\geq 0.05$.

In the remainder of the proof, we shall construct the prior measures $\mu_{0}$ and $\mu_{1}$ and give explicit values to the quantities $m$, $a_{m}$ and $M$ so that (74) is satisfied and $\Delta^{1/2}a_{m}M$ is the largest possible.

Step 3: Construction of the prior distributions $\mu_{0}$ and $\mu_{1}$.

We choose prior measures $\mu_{0}$ and $\mu_{1}$ such that the first moments of $\overline{\mu}_{0}$ and $\overline{\mu}_{1}$ are matching while $a_{m}$ and $M$ are as large as possible. The following lemma proved at the end of the subsection ensures the existence of such probability measure for a certain choice of $a_{m}$.

Lemma 3**.**

Given any positive and even integer $m$ and $p\in(0,1)$, define

[TABLE]

There exists two positive and symmetric measures $\nu_{0}$ and $\nu_{1}$ whose support lie in $[-1,-a_{m}]\cup[a_{m},1]$ satisfying:

[TABLE]

The implicit construction of $\nu_{0}$ and $\nu_{1}$ is based on a careful application of Hahn-Banach theorem together with extremal properties of Chebychev polynomials. It is inspired by the work of [28], but we go one step further to obtain the right dependency of $a_{m}$ with respect to $p$.

Fix $p=\overline{k}_{0}/\overline{k}_{1}$. Then, given $m\geq 1$, we consider the measures $\nu_{0}$ and $\nu_{1}$ as defined by Lemma 3 and the following remark. For $p=0$, we can define $\mu_{0}$ arbitrarily (take for instance $\mu_{0}=0.5\delta_{M}+0.5\delta_{-M}$). Given any measurable event $A$, we define $\mu_{0}$ (for $p\in(0,1)$) and $\mu_{1}$ by

[TABLE]

Note that $\mu_{0}$ and $\mu_{1}$ are symmetric and satisfy the support property (73) claimed at the beginning of the proof. Moreover, $\mu_{0}$ and $\mu_{1}$ have been defined in such a way that the moments of $\overline{\mu}_{0}$ and $\overline{\mu}_{1}$ are matching

[TABLE]

Step 4: Choice of $m$ and $M$.

In the sequel, we take $M^{2}:=m/(32e)$ and

[TABLE]

Equipped with this choice of parameters, we have

[TABLE]

where we used in the last line that $\Delta\leq k_{0}$ and $k_{0}\geq\sqrt{n}$ and $\delta\geq 0.05$. Hence, it suffices to the prove (74) to obtain (72).

Step 5: Control on the total variation distance between $\mathbf{P}_{0}$ and $\mathbf{P}_{1}$.

It remains to control the total variation distance between $\mathbf{P}_{0}$ and $\mathbf{P}_{1}$, relying on the fact that the $m$ first moments of $\overline{\mu}_{0}$ and $\overline{\mu}_{1}$ are matching. This is done in the following lemma.

Lemma 4**.**

The measures ${\mathbf{P}}_{0}$ and ${\mathbf{P}}_{1}$ satisfy

[TABLE]

as soon as $32eM^{2}\leq m$.

Although we take a slightly different path, the proof of this lemma is based on the same approach as in [10].

In the previous step, we have chosen $m$ in Equation (80) and $M$ in such a way that $\|{\mathbf{P}}_{0}-{\mathbf{P}}_{1}\|^{2}_{TV}\leq\exp(\delta^{2}/2)-1\leq\delta$. This concludes the proof that Equation (74) holds, and therefore that Equation (72) holds by Equation (80).

Step 6: Proof of (70) and (71)

Let us first prove (70). As explained earlier, a similar bound can be found in [3]. We elaborate on Le Cam’s approach. Let $M>0$ be a positive quantity that will be fixed later. We first define a suitable prior measure $\mu_{1}^{n}$ on the space $\mathbb{B}_{0}[\Delta,0,\Delta^{1/2}M]$.

Denote $\mathcal{S}(\Delta,n)$ the collection of all subset $S$ of $[n]$ of size $\Delta$. For any $S\in\mathcal{S}(\Delta,n)$, let $\mu_{1}^{S}$ denote the distribution of a vector $\theta$ where for all $i\in S$, $\theta_{i}\sim\frac{1}{2}\delta_{M}+\frac{1}{2}\delta_{-M}$ and for all $i\notin S$, $\theta_{i}$ follows a Dirac distribution at zero. As a consequence, $\mu_{1}^{n}:=\binom{n}{\Delta}^{-1}\sum_{S\in\mathcal{S}(\Delta,n)}\mu_{0}^{S}$ is a probability distribution over $\mathbb{B}_{0}[\Delta,0,\Delta^{1/2}M]$. Finally, we denote $\mathbf{P}_{1}:=\int\operatorname{\mathbb{P}}_{\theta}\mu_{1}^{n}(d\theta)$. Given a test $T$, its risk (5) is bounded

[TABLE]

As a consequence, the minimax separation distance $\rho^{*}_{\gamma}[n,0,\Delta]$ is larger than $\Delta^{1/2}M$, as soon as $\|\operatorname{\mathbb{P}}_{0}-\mathbf{P}_{1}\|_{TV}\leq\delta$, where $\delta:=1-\gamma\geq 0.5$. By Cauchy-Schwarz inequality,

[TABLE]

For fixed $S$ and $S^{\prime}$, the expectation

[TABLE]

When $|S|$ and $|S^{\prime}|$ are distributed uniformly in $\mathcal{S}(\Delta,n)$, the size $X:=|S\cap S^{\prime}|$ follows an hypergeometric distribution with parameters $n$, $\Delta$ and $\Delta/n$. We know from [1, p.173] that $X$ is distributed as the random variable $\operatorname{\mathbb{E}}[Z|\mathcal{B}_{n}]$ where $Z$ is a Binomial random variable with parameters $(\Delta,\Delta/n)$ and $\mathcal{B}_{n}$ is some $\sigma$-algebra. Applying Jensen inequality, we obtain

[TABLE]

Taking

[TABLE]

we conclude that $\|\operatorname{\mathbb{P}}_{0}-\mathbf{P}_{1}\|^{2}_{TV}\leq e^{\delta^{2}/2}-1\leq\delta^{2}$, implying that

[TABLE]

We have proved (70).

Finally, we turn to (71). Again, $M>0$ is a positive quantity that will be fixed later. Define $\theta_{1}$ as the constant vector whose components are all equal to $-M$. For any $S\in\mathcal{S}(\Delta,n)$, define $\theta_{0}^{S}$ the vector whose coordinates in $S$ are equal to zero and whose remaining components are equal to $-M$. Let $\mu_{0}^{n}:=\binom{n}{\Delta}^{-1}\sum_{S}\delta_{\theta_{0}^{S}}$. Finally, we denote $\mathbf{P}_{0}:=\int\operatorname{\mathbb{P}}_{\theta}\mu_{0}^{n}(d\theta)$. For any test $T$, $R[T,n-\Delta,n,\Delta^{1/2}M]\geq 1-\|\operatorname{\mathbb{P}}_{\theta_{1}}-\mathbf{P}_{0}\|_{TV}$ so that $\rho^{*}_{\gamma}[n,n-\Delta,\Delta]\geq\Delta^{1/2}M$ when $\|\operatorname{\mathbb{P}}_{\theta_{1}}-\mathbf{P}_{0}\|_{TV}\leq\delta$. Arguing as above, we get

[TABLE]

Choosing M^{2}=\log\big{[}1+\frac{\delta^{2}n}{2\Delta^{2}}\big{]}\geq\log\big{[}1+\frac{n}{8\Delta^{2}}\big{]}, we prove (71).

Proof of Proposition 6.

To derive Theorem 1, we only need to deduce from (70–71) the lower bounds in the regime (i) $k_{0}\leq\sqrt{n}$, (ii) $k_{0}>\sqrt{n}$ and $\Delta>k_{0}$ and (iii) $k_{0}>\sqrt{n}$ and $\Delta\leq 32\sqrt{(n-k_{0})\wedge k_{0}}$.

(i) $k_{0}\leq\sqrt{n}$. We combine (67) and (70) to obtain

[TABLE]

(ii) $k_{0}>\sqrt{n}$ and $\Delta>k_{0}$. We gather (68) and (72) to obtain

[TABLE]

(iii) $k_{0}>\sqrt{n}$ and $\Delta\leq 32\sqrt{(n-k_{0})\wedge k_{0}}$. We shall consider two subcases $\Delta\leq n^{1/3}$ and $\Delta>n^{1/3}$. For $\Delta\leq n^{1/3}$ and $k_{0}\leq n/2$, we apply (67) together with $\sqrt{n}/\Delta\geq n^{1/6}$.

[TABLE]

For $1\leq\Delta\leq n^{1/3}$ and $k_{0}>n/2$, we use (69) and (71).

[TABLE]

For $\Delta>n^{1/3}$ and $k_{0}\leq n/2$, we define $k^{\prime}_{0}:=\lfloor\Delta^{2}/(32)^{2}\rfloor$ and $n^{\prime}:=n-k_{0}+k^{\prime}_{0}$. Consequently, $\Delta>32\sqrt{k^{\prime}_{0}\wedge(n^{\prime}-k^{\prime}_{0})}$ and $k^{\prime}_{0}\geq\sqrt{n^{\prime}}$ for $n$ large enough. Then, (67) together with (72) gives us

[TABLE]

The last case $\Delta>n^{1/3}$ and $k_{0}>n/2$ is handled similarly.

∎

Proof of Lemma 3.

For the sake of clarity, we simply write $a$ for $a_{m}$ in this proof. Let $\mathcal{P}^{\text{sym}}_{m}$ denote the vector space of symmetric polynomials of degree smaller or equal to $m$. Define the linear function $g$ on $\mathcal{P}^{\text{sym}}_{m}$ by $g:P\mapsto P(0)$. We endow $\mathcal{P}^{\text{sym}}_{m}$ with the uniform norm $\|.\|_{[a,1]}$ on $[a,1]$. Let $\nu^{*}$ be the norm of this linear functional. By Hahn-Banach theorem, we can extend this functional from $\mathcal{P}^{\text{sym}}_{m}$ to the entire space $C[a,1]$ of continuous functions on $[a,1]$ without increasing the norm of the functional. By Riesz-Markov theorem, this linear functional can be represented as a measure $\nu$ on $[a,1]$. As a consequence, $\int P(t)\nu(dt)=P(0)$ for all $P\in\mathcal{P}^{\text{sym}}_{m}$ and the total variation $\|\nu\|_{TV}:=\int|\nu(dt)|$ equals $\nu^{*}$.

We extend $\nu$ to a symmetric measure $\bar{\nu}^{\text{sym}}$ on $[-1,-a]\cup[a,1]$ such that $\bar{\nu}^{\text{sym}}(\mathcal{A})=(\nu(\mathcal{A})+\nu(-\mathcal{A}))/2$. Let $\bar{\nu}^{\text{sym}}_{+}$ and $\bar{\nu}^{\text{sym}}_{-}$ respectively denote the positive and negative part of $\bar{\nu}^{\text{sym}}$ so that $\bar{\nu}^{\text{sym}}=\bar{\nu}^{\text{sym}}_{+}-\bar{\nu}^{\text{sym}}_{-}$. Finally, we define

[TABLE]

For any even integer $q\leq m$,

[TABLE]

where we used the symmetry of $t\mapsto t^{q}$ and the definition of the functional $g$ in the last equality. For any odd integer $q$

[TABLE]

because $q\mapsto t^{q}$ is antisymmetric. As a consequence, $\nu_{0}$ and $\nu_{1}$ satisfy the property (77). As for the measure of $\nu_{0}$ and $\nu_{1}$, we have

[TABLE]

by definition of $g$. Also

[TABLE]

by definition of $\nu^{*}$. As a consequence, $\int\nu_{1}(dt)=1$ and $\int\nu_{0}(dt)=\frac{\nu^{*}-1}{\nu^{*}+1}$. To conclude the proof of (76), we only need to show that

[TABLE]

Denote $\mathcal{P}_{m/2}$ the space of polynomials of degrees smaller or equal to $m/2$. We endow it with the supremum norm $\|.\|_{[a^{2},1]}$ on $[a^{2},1]$. Then the mapping $\phi:P(x)\mapsto P(x^{2})$ is an isometry from $(\mathcal{P}_{m/2},\|.\|_{[a^{2},1]})$ to $(\mathcal{P}_{m}^{\text{sym}},\|.\|_{[a,1]})$. Also, for $P\in\mathcal{P}_{m}^{\text{sym}}$, $P(0)=g(P)=[\phi^{-1}(P)](0)$. As a consequence, $\nu^{*}$ is characterized as

[TABLE]

Define the linear function $h:x\mapsto\frac{2}{1-a^{2}}t-\frac{1+a^{2}}{1-a^{2}}$ mapping $[a^{2},1]$ to $[-1,1]$. By substitution, we deduce that

[TABLE]

where we used the symmetry of the problem in the second identity. By Chebychev’s Theorem, this supremum is achieved by the Chebychev polynomial of order $m/2$. Hence, we get

[TABLE]

Since $\frac{1+\tanh^{2}(x)}{1+\cosh^{2}(x)}=\cosh^{2}(x)+\sinh^{2}(x)=\cosh(2x)$, we obtain $\nu^{*}=\tfrac{1+p}{1-p}$, which concludes the proof.

∎

Proof of Lemma 4.

By Cauchy-Schwarz inequality, we relate the total variation distance to the $\chi^{2}$ distance.

[TABLE]

Since ${\mathbf{P}}_{0}:=\otimes_{i}{\mathbf{P}}_{0,i}$ and ${\mathbf{P}}_{1}=\otimes_{i}{\mathbf{P}}_{1,i}$ are $n$-dimensional product measures. Developing the likelihood ratio, we arrive at

[TABLE]

So the two previous equations imply that

[TABLE]

We now focus on the $\chi^{2}$ distance $\int(d{\mathbf{P}}_{1,1}-d{\mathbf{P}}_{0,1})^{2}/d{\mathbf{P}}_{0,1}$. Recall that $k_{0}\geq\sqrt{n}$ and $m\geq 2$. We have by Equation (79) that

[TABLE]

where we used again $(a+b)^{2}\leq 2(a^{2}+b^{2})$. Since the function $x\mapsto\exp(-x)$ is convex, we can lower bound the density $d{\mathbf{P}}_{0,1}(y)/dy$ as follows

[TABLE]

where we used in the last line the symmetry of $\overline{\mu}_{0}$ and that its support lies in $[-M;M]$. Plugging the last inequality into Equation (84), we are equipped to bound the $\chi^{2}$ distance between ${\mathbf{P}}_{0,1}$ and ${\mathbf{P}}_{1,1}$.

[TABLE]

where we used the expression of $l$-th moments of a normal distribution in the second line. Now, assume that $32eM^{2}/m\leq 1$. Since $l!\geq(l/e)^{l}$, we have

[TABLE]

where we use in the two last line the $m\geq 2$ and $32eM^{2}/m\leq 1$. Coming back to (83), we conclude that, as soon as $32eM^{2}\leq m$,

[TABLE]

∎

B.2 Proofs of the testing upper bounds with known variance

B.2.1 Analysis of $T^{HC}_{\alpha,k_{0}}$

We will in fact prove a sharper result than Proposition 1. To study the rejection regions of this test, additional notation is needed. Given $\beta\in(0,1)$, let

[TABLE]

For any integer $q\in[1,n-k_{0}]$, define $t_{q}$

[TABLE]

and

[TABLE]

Proposition 7**.**

The type I error probability of $T^{HC}_{\alpha,k_{0}}$ is smaller or equal to $\alpha$, that is $\operatorname{\mathbb{P}}_{\theta}[T^{HC}_{\alpha,k_{0}}=1]\leq\alpha$ for all $\theta\in\mathbb{B}_{0}(k_{0})$. Besides, any $\theta\in\mathbb{R}^{n}$ such that

[TABLE]

belongs to the high probability rejection region of $T^{HC}_{\alpha,k_{0}}$, that is $\operatorname{\mathbb{P}}_{\theta}[T^{HC}_{\alpha,k_{0}}=1]\geq 1-\beta$.

Proposition 1 is a straightforward corollary of Proposition 7.

Proof of Proposition 7.

We first focus on the type I error and then consider the power of the procedure.

Level of the Test. Consider any $\theta\in\mathbb{B}_{0}(k_{0})$. For any $t>0$, $N_{t}-k_{0}$ is stochastically smaller than a Binomial distribution with parameters $n-k_{0}$ and $2\Phi(t)$. Since $\Phi(t)\leq\exp(-t^{2}/2)$, we obtain by a simple union bound that

[TABLE]

Also, by Bernstein inequality, we have

[TABLE]

Applying again an union bound yields

[TABLE]

Power of the test. To ease the notation, we respectively write $\mu_{q}$, $u_{q}$ and $q_{+}$ for $\mu^{HC}_{q}$, $u_{t_{q},k_{0}}^{HC}$ and $q_{+}^{HC}$. Let $\theta$ be any vector such that $|\theta_{(k_{0}+q)}|\geq\mu$. The proof is divided into two different cases depending on the value of $q$.

Case 1: Assume that $q<q_{+}$. In that situation, we focus on $N_{t^{HC}_{*,\alpha}}$:

[TABLE]

Restricting ourselves to the $k_{0}+1$ largest absolute values of $\theta$, we get

[TABLE]

which is smaller than $\beta$, since by definition (87), we have $\mu_{q}\geq t^{HC}_{*,\alpha}+\sqrt{2\log[(k_{0}+1)/\beta]}$.

Case 2: We now assume that $q\geq q_{+}$. By definition (85) and (86) of $q_{+}$ and $t_{q}$, this enforces that $t_{q}<t^{HC}_{*,\alpha}$. Observe that $N_{t_{q}}$ is stochastically larger than the sum of a Binomial random variable with parameters $(k_{0}+q)$ and $(1-\Phi(\mu_{q}-t_{q}))$ and a Binomial random variable with parameters $(n-k_{0}-q)$ and $2\Phi(t_{q})$. Applying Bernstein inequality to these two random variables, we derive that, with probability larger than $1-\beta$,

[TABLE]

where

[TABLE]

As a consequence of (11), we have $\operatorname{\mathbb{P}}_{\theta}[T^{HC}_{\alpha,k_{0}}=1]\geq 1-\beta$ as soon as

[TABLE]

Since $t_{q}\geq 1$ and $\mu_{q}-t_{q}\geq 2$, we have $2\Phi(t_{q})\leq 0.4$ and $\Phi(\mu_{q}-t_{q})\leq 0.1$. As a consequence, the above inequality holds when the four following conditions are satisfied

[TABLE]

The last condition is a consequence of the condition $q\geq q_{+}$. To finish the proof, it suffices to show that (88), (89), and (90) are ensured by our choice of $\mu_{q}$ and $t_{q}$. Inequalities (88) and (89) hold when

[TABLE]

In view of the definition (87) of $\mu_{q}$, this last inequality is true. Since $(\sqrt{x+y}+\sqrt{z})^{2}\leq(2+x)(1+y+z)$, Condition (90) holds if

[TABLE]

Since $\Phi(t_{q})\leq\tfrac{e^{-t_{q}^{2}/2}}{t_{q}\sqrt{2\pi}}$ and $t-\log(t)-1>0$ for any $t>0$, we only need that

[TABLE]

which is a straightforward consequence of our choice (86) of $t_{q}$. This concludes the proof.

∎

B.2.2 Analysis of $T^{B}_{\alpha,k_{0}}$

To properly characterize the power of $T^{B}_{\alpha,k_{0}}$ additional notation is needed. Let

[TABLE]

For any integer $q>4v_{k_{0}}^{B}$ define

[TABLE]

Proposition 8**.**

The type I error probability of $T^{B}_{\alpha,k_{0}}$ is smaller or equal to $\alpha$, that is $\operatorname{\mathbb{P}}_{\theta}[T^{B}_{\alpha,k_{0}}=1]\leq\alpha$ for all $\theta\in\mathbb{B}_{0}(k_{0})$. Besides, any $\theta\in\mathbb{R}^{n}$ such that any of the two following conditions is fulfilled

[TABLE]

belongs to the high probability rejection region of $T^{B}_{\alpha,k_{0}}$, that is $\operatorname{\mathbb{P}}_{\theta}[T^{B}_{\alpha,k_{0}}=1]\geq 1-\beta$.

Proof of Proposition 2.

Proposition 2 is a simple consequence of Proposition 8 based on the fact that $v_{k_{0}}^{B}\geq c_{\alpha,\beta}\sqrt{n}$ and $s_{k_{0}}=1$ for $k_{0}<\sqrt{n}$ whereas $s_{k_{0}}\geq c\log(1+k_{0}/\sqrt{n})$ for $k_{0}\geq\sqrt{n}$. ∎

Proof of Proposition 8.

To ease the notation, we respectively write $v_{k_{0}}$, $\mu_{q}$ and $s$ for $v_{k_{0}}^{B}$, $\mu_{q}^{B}$ and $s_{k_{0}}$. The proof is divided into two main lemmas. First, we prove that $Z(s)$ concentrates well around its expectation using the Gaussian concentration Theorem.

Lemma 5**.**

For any $x>0$ and any $\theta\in\mathbb{R}^{n}$ and any $s>0$, it holds

[TABLE]

Note that Hoeffding’s inequality allows to recover a similar inequality with a less stringent dependency with respect to $s$.

In view of the above deviation inequality, it suffices to control the expectations of $\operatorname{\mathbb{E}}_{\theta}[Z(s)]$ to derive the type I and type II error probabilities. When $X\sim\mathcal{N}(\mu,1)$, the expectation of $\kappa_{s}(X)$ satisfies

[TABLE]

Define the function $g$ by $g(0)=0$ and $g(x):=1-2\frac{1-\cos(x)}{x^{2}}$ for $x\neq 0$. We have

[TABLE]

Since $\cos(x)\geq 1-x^{2}/2$, $g$ takes values in $[0,1]$.

Level of the Test. Consider any $\theta\in\mathbb{B}_{0}(k_{0})$. Since $g$ takes values in $[0,1]$ and since $g(0)=0$, we have $\operatorname{\mathbb{E}}_{\theta}[Z(s)]\leq k_{0}$. Gathering this bound with the deviation inequality (95) and the definition of $s$, we conclude that

[TABLE]

In view the definition (17) of $T_{B}$, this implies that $\operatorname{\mathbb{P}}_{\theta}[T^{B}_{k_{0}}=1]\leq\alpha$.

Power of the test. Turning to the type II error, we first consider any vector $\theta$ satisfying Condition (93). Applying again the deviation inequality (95) together with (97), we have

[TABLE]

Observe that

[TABLE]

Hence, the error probability $\operatorname{\mathbb{P}}_{\theta}[T^{B}_{\alpha,k_{0}}=0]$ is smaller than $\beta$ as soon as

[TABLE]

where $v_{k_{0}}$ is defined in (91). Thus, it suffices to prove (98). The control of the expectation $\operatorname{\mathbb{E}}_{\theta}[Z(s)]=\sum g(s\theta_{i})$ is more challenging than under the null hypothesis.

Observe that, for large $x$, $g(x)$ goes to one at rate $1/x^{2}$. For small $x$, a Taylor expansion of $\cos(x)$ leads to $g(x)=O(x^{2})$. Let us provide non-asymptotic lower bounds of $g$ matching these two asymptotic behaviors. Since $1-\cos(x)\leq 2$, it follows from the definition of $g$ that $g(x)\geq 1-4/x^{2}$ for any $x\neq 0$. Studying the three first derivatives of $g$, we derive that $g$ is increasing on $[0,\pi]$. By Taylor’s Lagrange formula, $g(x)\geq\frac{x^{2}}{6}\cos(\pi/3)$ for any $x\in[0,\pi/3]$. Since $g$ is increasing on $[0,\pi]$, this implies that $g(x)\geq(\frac{\pi}{3\cdot 2.1})^{2}\cdot\frac{x^{2}}{12}\geq x^{2}/50$ for any $x\in[0,2.1]$. We have proved that

[TABLE]

Observe that the function $f$ defined by $f(x)=x^{2}/50$ if $|x|\leq 2.1$ and $f(x)=1-4/x^{2}$ for $x>2.1$ is increasing with respect to $|x|$. Since $g(x)$ is non-negative for all $x$, it follows from Condition (93) that the two following inequalities hold

[TABLE]

We consider two cases.

Case 1: $q\leq\frac{256(k_{0}+v_{k_{0}})}{(2.1)^{2}}$. By Definition (92) of $\mu_{q}$, we have $(s\mu_{q})^{2}/4=64(k_{0}+v_{k_{0}})/q$. The above condition on $q$ enforces that $62(k_{0}+v_{k_{0}})\geq q$, which in turn implies that $q(s\mu_{q})^{2}/4-q\geq 2k_{0}$. Hence, we have

[TABLE]

Also, the condition on $q$ enforces that $\frac{4}{(s\mu_{q})^{2}}\leq(2/2.1)^{2}$. Since we assume in (93) that $q\geq 2/[1-(2/2.1)^{2}]v_{k_{0}}$, it follows that

[TABLE]

Then, the lower bound (101) implies that

[TABLE]

where we used $x\vee y\geq(x+y)/2$. We have proved (98).

Case 2: $q>\frac{256(k_{0}+v_{k_{0}})}{(2.1)^{2}}$. This implies that $s\mu_{q}\leq 2.1$ allowing us to apply the lower bound (100). Thus,

[TABLE]

It remains to prove that the power of $T^{B}_{\alpha,k_{0}}$ is larger than $1-\beta$ for any $\theta$ satisfying (94). From the lower bound (99) (and since the function $f$ derived from this lower bound (99) is increasing with respect to $|x|$), we deduce that

[TABLE]

which implies (98). This concludes the proof.

∎

Proof of Lemma 5.

As a matter of fact, $Z(s)=f(Y_{1},\ldots,Y_{n})$ is a lipschitz function of the Gaussian vector $Y_{1},\ldots Y_{n}$. In order to apply the Gaussian concentration theorem, we need to bound its lipshitz norm $\|f\|_{L}$. The derivative $\kappa^{\prime}_{s}(x)$ of $\kappa_{s}(x)$ satisfies

[TABLE]

As a consequence, $\|f\|_{L}\leq\frac{2}{s}\sqrt{n}e^{s^{2}/2}$, which concludes the proof. ∎

B.2.3 Analysis of $T_{\alpha,k_{0}}^{I}$

For any $l\in\mathcal{L}_{k_{0}}$, let

[TABLE]

Define $q^{I}_{\min}:=16l_{k_{0}}+4v^{I}_{k_{0},l_{k_{0}}}$. For any integer $q\geq q^{I}_{\min}$, let

[TABLE]

Proposition 9**.**

Assume that $k_{0}\geq 20\sqrt{n}$ and that $n$ is large enough. The type I error probability of $T^{I}_{\alpha,k_{0}}$ is smaller or equal to $\alpha$. Besides, any $\theta\in\mathbb{R}^{n}$ such that

[TABLE]

belongs to the high probability rejection region of $T^{I}_{\alpha,k_{0}}$, that is $\operatorname{\mathbb{P}}_{\theta}[T^{I}_{\alpha,k_{0}}=1]\geq 1-\beta$.

Proposition 3 is a straightforward corollary of the above proposition. Indeed, we have $q_{\min}^{I}\leq c_{\alpha,\beta}\sqrt{k_{0}n^{1/2}}$. Since $l(q)\geq l_{k_{0}}=\lceil\sqrt{k_{0}n^{1/2}}\rceil$ and $k_{0}\geq 5\sqrt{n}$, it follows that $\log(l(q)/\sqrt{n})\geq\log(l_{k_{0}}/\sqrt{n})\geq c\log(1+k_{0}/\sqrt{n})$. For any we have $l\in\mathcal{L}_{k_{0}}$, $v_{k_{0},l}^{I}\leq c_{\alpha,\beta}l$ implying that $l(q)\geq c_{\alpha,\beta}[q\wedge k_{0}]$ and $\mu_{q}^{I}\leq c_{\alpha,\beta}\frac{1+\log(1+k_{0}/q)}{\sqrt{\log(1+k_{0}/\sqrt{n})}}$.

Before proving Proposition 9, we start with a deviation inequality inequality for $V(r_{k_{0},l},w_{l})$.

Lemma 6**.**

For any $\theta\in\mathbb{R}^{n}$, any $k_{0}\geq 20\sqrt{n}$ and any $l\in\mathcal{L}_{k_{0}}$ and any $x>0$, we have

[TABLE]

Proof of Lemma 6.

Fix $\theta\in\mathbb{R}^{n}$ and $l\in\mathcal{L}_{k_{0}}$. Then $n-V(r_{k_{0},l},w_{l})=\sum_{i}\eta_{r_{k_{0},l},w_{l}}(Y_{i})$ is a sum of $n$ independent random variables bounded in absolute values by

[TABLE]

where we used that $l\leq k_{0}/4$. Then, Hoeffding’s inequality yields

[TABLE]

∎

Proof of Proposition 9.

To ease the notation, we respectively write $v_{l}$, $\mu_{q}$ and $r_{l}$ for $v_{k_{0},l}^{I}$, $\mu_{q}^{I}$, and $r_{k_{0},l}$. We start with a few simple observations that will be used multiple times.

Lemma 7**.**

For any $l\in\mathcal{L}_{k_{0}}$, we have $r_{l}\geq\sqrt{2\log(4)}\geq 1$, $(1-2\Phi(r_{l}))\geq 0.65$ and $r_{l}\leq\sqrt{2}w_{l}$.

Proof of Lemma 7.

Since for all $l\in\mathcal{L}_{k_{0}}$, $l\leq k_{0}/4$, $r_{l}\geq\sqrt{2\log(4)}\geq 1$. Computing the quantile of a standard normal distribution, we obtain $1-2\Phi(1)\geq 0.65$. For all $l\in\mathcal{L}_{k_{0}}$, we have $l^{2}\geq l_{k_{0}}^{2}\geq k_{0}\sqrt{n}$, which implies $r_{l}\leq\sqrt{2}w_{l}$. ∎

Let us now consider the expectation of the statistic $V(r_{l},w_{l})$. Given this alternative expression of $\eta_{r,w}(x)$,

[TABLE]

we get, for $X\sim\mathcal{N}(x,1)$,

[TABLE]

In the sequel, we denote $\Psi_{l}(x)$ this expectation. Obviously,

[TABLE]

whereas $\Psi_{l}(x)=1$ if and only if $x=0$. The following lemma states sharper bounds for $\Psi_{l}(x)$.

Lemma 8**.**

For any $x\in\mathbb{R}$,

[TABLE]

As Lemmas (8) and 6 provide controls on both the expectation and the deviation of $V(r_{l},w_{l})$, we are equipped to bound the type I and type II error probabilities of $T_{\alpha,k_{0}}^{I}$.

Level of the Test. Consider any $\theta\in\mathbb{B}_{0}(k_{0})$. Since $\Psi_{l}(0)=1$,

[TABLE]

where we used Lemma 8. Applying the deviation inequality (105) to $V(r_{l},w_{l})$ with the weight $\log(\frac{\pi^{2}[1+\log_{2}(l/l_{k_{0}})]^{2}}{6\alpha})$, we derive that, with probability larger than $1-\frac{6\alpha}{\pi^{2}[1+\log_{2}(l/l_{k_{0}})]^{2}}$,

[TABLE]

Since $\sum_{l\in\mathcal{L}_{k_{0}}}\frac{1}{[1+\log_{2}(l/l_{k_{0}})]^{2}}\leq\sum_{i=1}^{\infty}i^{-2}=\pi^{2}/6$, it follows that (110) is simultaneously valid for all $l\in\mathcal{L}_{k_{0}}$ with probability larger than $1-\alpha$. Together with (109), this implies that the size of the test of $T_{\alpha,k_{0}}^{I}$ is smaller than $\alpha$.

Power of the Test. Let us now consider any vector $\theta$ satisfying (104). We take $q\geq q_{\min}^{I}$ such that $|\theta_{(k_{0}+q)}|\geq\mu_{q}$. In the sequel, we simply write $l$ for $l(q)$. Using (108) together with the bound $\Psi_{l}(x)\leq 1$ we obtain

[TABLE]

where we used the definition (104) of $\mu_{q}$ and $k_{0}\geq 4l_{\max}\geq 4l$ in the last line. Together with the deviation inequality (105), we obtain

[TABLE]

Coming back to the definition (103) of $l$, this implies that $T_{\alpha,k_{0}}^{I}$ rejects the null hypothesis with probability larger than $1-\beta$.

∎

Proof of Lemma 8.

If we replace the integral of $[-r_{l},r_{l}]$ by an integral over $\mathbb{R}$ in the definition (107) of $\Psi_{l}(x)$, we recognize the Fourier transform of a standard normal variable.

[TABLE]

Denote $\vartheta_{l}(x):=\int_{r_{l}}^{\infty}\phi(\xi)\cos(\xi\tfrac{w_{l}}{r_{l}}x)d\xi$ the remainder term.

Let $\bar{r}_{l}\geq r_{l}$ be the smallest number satisfying $\bar{r}_{l}\equiv\pi/2[\pi]$. Since the function $\xi\mapsto\phi(\xi)$ is decreasing on $[r_{l},\infty)$, the integral in $\vartheta_{l}(x)$ can be decomposed as an alternative sum

[TABLE]

where the sign of the integral over $[r_{l},\bar{r}_{l}]$ is opposite to the one over $[\bar{r}_{l},\bar{r}_{l}+r_{l}\pi/(w_{l}x)]$. As a consequence,

[TABLE]

Coming back to the decomposition of $\Psi_{l}(x)$, we obtain

[TABLE]

where we used Lemma 7 in the last inequality. Since $1-2\Phi(r_{l})$ is larger than $1/2$ (Lemma 7 again), the above inequality implies (108).

∎

B.2.4 Analysis of $T^{C}_{\alpha,k_{0}}$

Proof of Corollary 1.

The first bound is a straightforward consequence of Propositions 1, 2, and 3. We focus on the second bound (25). Choosing the constant $c^{\prime}_{\alpha,\beta}$ small enough, we claim that Condition (25) implies that either Condition (24) is true for some $q\leq\Delta$ or that (19) is true. Corollary 1 is then a straightforward consequence of this claim.

We will prove this claim by contraposition. In the sequel, we assume that both (19) and (24) for all $q\leq\Delta$ are not satisfied. The analysis is divided into 5 cases depending on the values of $k_{0}$, $\Delta$ and $n$.

Case A.1: $k_{0}\leq\sqrt{n}$ and $\Delta\leq\sqrt{n}$. We consider two subcases: (i) $\Delta\leq\sqrt{n}/2$ and (ii) $\Delta>\sqrt{n}/2$. In case (i), the fact that Condition (24) is not satisfied implies

[TABLE]

which contradicts (25). In case (ii), $\log(1+\sqrt{n}/\Delta)\geq\log(2)$. For $n$ large enough, $\sqrt{n}/\lfloor\sqrt{n}/2\rfloor\geq 3$. Using the above bound, we get

[TABLE]

which contradicts again (25) if $c^{\prime}_{\alpha,\beta}$ in (25) is chosen small enough.

Case A.2: $k_{0}\leq\sqrt{n}$ and $\Delta\geq\sqrt{n}$. We start from

[TABLE]

The first sum is small in front of $\sqrt{n}$ by Case A.1(i). Since (24), is not not satisfied this implies that all $|\theta_{(k_{0}+q)}|$ for $q\geq\sqrt{n}$ are (up to multiplicative constants) smaller than $1=s_{k_{0}}$. Together with the fact that (19) is not satisfied, this implies that

[TABLE]

We have proved that

[TABLE]

which contradicts (25) if $c^{\prime}_{\alpha,\beta}$ in (25) is chosen small enough.

Case B.1: $k_{0}>\sqrt{n}$ and $\Delta\leq\sqrt{n}$. We argue exactly as in case A.1(i). The fact that Condition (24) is not satisfied implies that

[TABLE]

which contradicts (25) if $c^{\prime}_{\alpha,\beta}$ in (25) is chosen small enough.

Case B.2: $k_{0}>\sqrt{n}$ and $\sqrt{n}<\Delta\leq k_{0}$. The fact that Condition (24) is not satisfied implies

[TABLE]

Let us focus on the last sum in the rhs. Comparing the sum with an integral yields

[TABLE]

Putting everything together, we obtain

[TABLE]

which contradicts (25) if $c^{\prime}_{\alpha,\beta}$ in (25) is chosen small enough.

Case B.3: $k_{0}>\sqrt{n}$ and $\Delta>k_{0}$. As in Case A.2, we divide the distance into two sums.

[TABLE]

The first sum has already been handled in Case B.2 and is (up to constants) smaller than $k_{0}/\log[1+k_{0}/\sqrt{n}]$. Condition (24) ensures that all coefficients $\theta_{(k_{0}+q)}$ with $q>k_{0}$ are (in absolute values and up to constants) smaller than $1/\log[1+k_{0}/\sqrt{n}]$. As a consequence,

[TABLE]

which is (up to constants) smaller than $k_{0}/\log[1+k_{0}/\sqrt{n}]$ by Condition (19). This contradicts again (25) if $c^{\prime}_{\alpha,\beta}$ in (25) is chosen small enough.

∎

Appendix C Proof of Theorem 2

As in the previous section, it is assumed that $\sigma=1$. These proofs follow closely the same steps as the analysis of $T^{HC}_{\alpha,k_{0}}$, $T^{B}_{\alpha,k_{0}}$, and $T^{I}_{\alpha,k_{0}}$ in Section B.2. We also use the same notation. Fix any $\theta\in\mathbb{R}^{n}$.

Type I error (Proof of (30)). We consider separately $\widehat{k}^{HC}$, $\widehat{k}^{B}$ and $\widehat{k}^{I}$ and we will prove that, for each of them, the probability that it exceeds $\|\theta\|_{0}$ is smaller than $\alpha/3$. First, we consider $\widehat{k}^{HC}$. Arguing as in the proof of Proposition 7, we have

[TABLE]

For any positive integer $t$, $N_{t}-\|\theta\|_{0}$ is stochastically larger than a Binomial distribution with parameter $(n-\|\theta\|_{0},2\Phi(t))$. In view of the definition 12 of $u_{t,\alpha}^{HC}$, Bernstein’s inequality yields

[TABLE]

Taking an union bound over all $t\in\mathcal{T}$, we derive that with probability larger than $1-\alpha/6$,

[TABLE]

We have proved that $\operatorname{\mathbb{P}}_{\theta}[\widehat{k}^{HC}>\|\theta\|_{0}]\leq\alpha/3$.

Let us turn $\widehat{k}^{B}$. Lemma 5 provides a deviation inequality for all statistics $Z(s)$. Together with the definition (17) of $u_{k_{0},\alpha}^{B}$, this yields

[TABLE]

for all $k_{0}$ in the dyadic collection $\mathcal{K}_{0}$. Besides the identity (96) ensures that $\operatorname{\mathbb{E}}_{\theta}[Z(s)]\leq\|\theta\|_{0}$ for any $s>0$. Taking an union bound over all $k_{0}\in\mathcal{K}_{0}$, we obtain $\operatorname{\mathbb{P}}_{\theta}[\widehat{k}_{B}\leq k_{0}]\geq 1-\alpha/3$.

Finally, we consider $\widehat{k}^{I}$. The deviation inequality for $V(r,w)$ (Lemma 6) and the definition (22) of $u^{I}_{k_{0},l,\alpha_{k_{0}}}$ ensures that, with probability larger $1-\alpha_{k_{0}}$, we have

[TABLE]

simultaneously for all $l\in\mathcal{L}_{k_{0}}$. By Lemma 8, we have $\operatorname{\mathbb{E}}_{\theta}[V(r_{k_{0},l},w_{l})]\leq\|\theta\|_{0}(1+\frac{l}{k_{0}})$. Since $\sum_{k_{0}\in\mathcal{K}_{0}}\alpha_{k_{0}}\leq\alpha/3$, we conclude that $\operatorname{\mathbb{P}}_{\theta}[\widehat{k}_{I}\leq\|\theta\|_{0}]\geq 1-\alpha/3$.

Type II error (Proof of (31) and (32)). For any $t>0$, we denote $N_{t}^{\theta}$ the number of components of $\theta$ larger or equal to $t$ (in absolute value). Also, we write $t_{*,\alpha}$ for $t_{*,\alpha}^{HC}$. Arguing as for the type I error, we shall prove that with probability larger than $1-\beta$, all the statistics involved in $\widehat{k}_{HC}$, $\widehat{k}_{B}$, and $\widehat{k}_{I}$ are not much smaller than their expectation. First, an union bound tell us that, with probability larger than $1-\beta/6$,

[TABLE]

Besides, for any $t>0$, $N_{t}$ is stochastically larger than a sum of a Binomial distribution with parameter $(N_{2t}^{\theta},1-\Phi(t))$ and Binomial distribution with parameter $(n-N_{2t}^{\theta},2\Phi(t))$. Since the variance of this sum is smaller than $2n\Phi(t)$, it follows from Bernstein’s inequality together with an union bound that, with probability larger than $1-\beta/6$, we have

[TABLE]

simultaneously for all $t\in\mathcal{T}$. For any $k_{0}\in\mathcal{K}_{0}$, denote $\beta_{k_{0}}:=2\beta([1+\log_{2}(\tfrac{k_{0}}{k_{\min}})]^{2}\pi^{2})^{-1}$. Then, Lemmas 5 and 6, ensure that, with probability larger than $1-2\beta/3$,

[TABLE]

simultaneously for all $k_{0}\in\mathcal{K}_{0}$ and all $l\in\mathcal{L}_{k_{0}}$. Putting everything together, we conclude that with probability larger than $1-\beta$, we have $\widehat{k}\geq k^{\theta}_{HC}\vee k^{\theta}_{B}\vee k^{\theta}_{I}$, where these three deterministic quantities are defined by

[TABLE]

We study separately the consequence of the three inequalities $\widehat{k}\geq k^{\theta}_{HC}$, $\widehat{k}\geq k^{\theta}_{B}$, and $\widehat{k}\geq k^{\theta}_{I}$. First, we consider $k^{\theta}_{HC}$. Define q_{+}:=\frac{16}{3}\log\big{(}\frac{t_{*,\alpha/3}^{2}\pi^{2}}{3(\alpha\wedge\beta)}\big{)} and fix any $q\in[n-\widehat{k}]$.

Case 1: $q\leq q_{+}$. The condition $k^{\theta}_{HC}\leq\widehat{k}$ implies $N^{\theta}_{t_{*,\alpha/3}+t_{*,\beta/3}}\leq\widehat{k}<\widehat{k}+q$, which is equivalent to

[TABLE]

since $q\leq q_{+}$.

Case 2: $q>q_{+}$. Let $t$ be the smallest number such that 8\Phi(t)<1\vee\frac{q}{\widehat{k}}\vee\frac{q^{2}}{32n\log\big{(}\frac{t^{2}\pi^{2}}{3(\alpha\wedge\beta)}\big{)}}. Then, we take $t^{\prime}=\lceil t\rceil\wedge t_{*,\alpha/3}$. If $t^{\prime}<t_{*,\alpha/3}$, we have $N^{\theta}_{2t^{\prime}}<\widehat{k}+q$. Indeed, $N^{\theta}_{2t^{\prime}}\geq\widehat{k}+q$ would imply

[TABLE]

where we used the definition of $t$ and that $q>q_{+}$. This contradicts $\widehat{k}\geq k^{\theta}_{HC}$. We have proved that \big{|}\theta_{(\widehat{k}+q)}\big{|}\leq c_{\alpha,\beta}\big{[}1+\sqrt{\log(1+\frac{\widehat{k}\vee\sqrt{n}}{q})}\big{]}. If $t^{\prime}=t_{*,\alpha/3}$, then we have $N_{t^{\prime}+t_{*,\beta/3}}^{\theta}\leq\widehat{k}<\widehat{k}+q$ as in Case 1. Gathering the bounds for Cases 1 and 2, we have proved that, for all $q=1,\ldots,n-\widehat{k}$,

[TABLE]

Turning to $k^{\theta}_{B}$, we define $\overline{k}_{0}$ as the smallest $k_{0}\in\mathcal{K}_{0}$ such that $k_{0}\geq\widehat{k}/2$. Note that $\overline{k}_{0}$ always exists since $k_{\max}>n/2$. The definition (116) of $k^{\theta}_{B}$ implies that

[TABLE]

where we used in the second line the definition of $\alpha_{k_{0}}$ and of $u^{B}_{\overline{k}_{0},\alpha_{k_{0}}}$ and $\overline{k}_{0}\geq k_{\min}\geq\sqrt{n}$ in the last line. From the definition (96) of the expectation of $Z(s_{\overline{k}_{0}})$ and its lower bound (99), we derive that

[TABLE]

since $g$ is increasing. As a consequence,

[TABLE]

Relying on the definition (99) of $f$, we obtain

[TABLE]

Finally, we investigate $k^{\theta}_{I}$. Since (118) and (120) are alone sufficient to prove (31) for $\widehat{k}\leq 40\sqrt{n}$. We assume henceforth that $\widehat{k}\geq 40\sqrt{n}$. Let $\overline{k}_{0}$ be defined as previously. Note that $\overline{k}_{0}$ is larger than $20\sqrt{n}$. The definition (117) of $k^{\theta}_{I}$ implies that, for all $l\in\mathcal{L}_{\overline{k}_{0}}$,

[TABLE]

where we used the definition (22) of $u^{I}_{k_{0},l,\alpha}$ in the second line and the inequalities $\overline{k}_{0}\geq k_{\min}\geq\sqrt{n}$, $l\geq\sqrt{n^{1/2}\overline{k}_{0}}$ in the third line. Lemma 8 then ensures that

[TABLE]

since $l\leq\overline{k}_{0}/4$ by definition of $\mathcal{L}_{\overline{k}_{0}}$. These two bounds imply that for all $q\geq 1$ and all $l\in\mathcal{L}_{\overline{k}_{0}}$, we have

[TABLE]

with the convention $\log(1/0)=\infty$. For any $q\geq 2c^{\prime}_{\alpha,\beta}l_{\overline{k}_{0}}\geq c^{\prime}_{\alpha,\beta}\sqrt{2\widehat{k}n^{1/2}}$ with $c^{\prime}_{\alpha,\beta}$ as above, we obtain by taking $l_{q}=\max\{l\in\mathcal{L}_{k_{0}},\ \text{such that }q\geq 2c^{\prime}_{\alpha,\beta}\}$, that

[TABLE]

Putting together (118), (120) and (121) and playing with the constants, we prove (31).

As argued in the proof of Corollary 1, the second result (32) is a consequence of (31) together with the upper bound.

[TABLE]

Thus, we will skip the details for (32) and only prove (122). Starting from (119) and the expression (96) of $\operatorname{\mathbb{E}}_{\theta}[Z_{\overline{k}_{0}}]$. We have

[TABLE]

By (99), the function $g$ satisfies $g(x)\geq c(x^{2}\wedge 1)$. Since $s^{2}_{\overline{k}_{0}}\geq s^{2}_{\widehat{k}}-\log(2)\geq cs^{2}_{\widehat{k}}$, it follows that

[TABLE]

which implies (122).

Proof of Corollary 4.

The first negative result (36) is a consequence of the minimax lower bounds in Section 2. The second negative (37) result is expressed in terms of the tail distribution of $\theta$ rather in terms of its $l_{2}$ distance to a sparsity ball. Nevertheless, one may readily adapt all the proofs of the testing minimax lower bounds to account for this modification. ∎

Appendix D Proofs of the results with unknown variance

D.1 Proof of the lower bounds

D.1.1 Proof of Proposition 4

This proposition is mostly a consequence of other results in this manuscript. When $\Delta\geq\sqrt{n}$, the minimax lower bound is a consequence of Theorem 1 for known variance. The extension of the Higher criticism statistic to unknown variance as described in Section 4.3 below achieves the matching upper bound as proved in Theorem 5. For $\Delta\geq\sqrt{n}$, the lower bound (43) is a consequence of Theorem 3. To prove the minimax upper bound in (43), we rely on the statistic $S_{4}=\frac{n\|Y\|_{4}^{4}}{\|Y\|_{2}^{2}}-3$ defined in (44). Under the null, Chebychev inequality enforces that $\|Y\|_{4}^{4}/\sigma^{3}=3n+O_{P}(\sqrt{n})$ and that $\|Y\|_{2}^{2}/\sigma^{2}=n+O_{P}(\sqrt{n})$. As a consequence, $S_{4}=O_{P}(1/\sqrt{n})$. Under the alternative, one has

[TABLE]

so that

[TABLE]

where we used $\|\theta\|_{2}^{4}\leq\|\theta\|_{0}\|\theta\|_{4}^{4}\leq\Delta\|\theta\|_{4}^{4}$. Besides, for $\|\theta\|_{4}^{4}/\sigma^{4}\geq\sqrt{n}$, one has $\|\theta\|_{6}^{3}/\sigma^{3}\leq\sqrt{n}+2\|\theta\|_{4}^{4}/\sigma^{4}n^{-1/8}$ (consider separately the components of $\theta$ smaller than one 1, between $1$ and $n^{1/8}$ and larger than $n^{1/8}$). As a consequence, if $\|\theta\|_{4}^{4}/\sigma^{4}$ is large enough is front of $\sqrt{n}$, then $S_{4}$ will be also large in front of $\sqrt{n}$ with high probability. Define a test $T_{4}$ rejecting for large values of $S_{4}$ in such a way that the size of $T_{4}$ is equal to $\gamma/2$. It follows from the above discussion that the type II error probability will be smaller than $\gamma/2$ for $\|\theta\|_{4}^{4}\geq c_{\gamma,\eta}\sigma^{4}\sqrt{n}$. Since Cauchy-Schwarz inequality enforces that $\|\theta\|_{2}^{2}\leq\sqrt{\Delta}\|\theta\|_{4}^{2}$, this implies that $\rho_{\gamma,\mathrm{var}}^{*2}[T_{4};0,\Delta]\leq c^{\prime}_{\gamma,\eta}\sigma_{+}^{2}\sqrt{\Delta n^{1/2}}$, which concludes the proof.

D.1.2 Proof of Theorem 3

By homogeneity, we assume that $\sigma_{-}\leq 1\leq\sigma_{+}\leq 2$ in this proof.

Case 1 : $k_{0}=0$.

Let us first consider the case $k_{0}=0$. This proof follows the same general approach as that of Theorem 1 for $k_{0}=0$. Fix $\Delta^{\prime}=\Delta/2$. Define the probability measures $\mu_{0}=\delta_{0}$ and $\mu_{1}=\frac{\Delta^{\prime}}{2n}(\delta_{-M}+\delta_{M})+\frac{(1-\Delta^{\prime})}{n}\delta_{0}$, where $M^{8}=\Upsilon\frac{n}{(\Delta^{\prime})^{2}}$ where $\Upsilon\leq 1$ is a positive constant to be fixed. In the sequel, we denote $p=\Delta^{\prime}/(2n)$ and $v^{2}=2pM^{2}$. Finally, we define

[TABLE]

Note that, when $Y\sim\mathbf{P}_{1}$, the marginal variance $\operatorname{Var}\left(Y_{i}\right)$ are all equal to one.

Let $\theta$ be sampled according to the product distribution $\mu_{1}^{\otimes n}$. Since $\Delta\geq\sqrt{n}$, Bernstein’s inequality implies that $\mu_{1}^{\otimes n}[\|\theta\|_{0}\in[\Delta/4,\Delta]$ is close to one (and in particular is larger than 0.55). As in the proof of Theorem 1 (Step 2), if we can prove that $\|\mathbf{P}_{0}-\mathbf{P}_{1}\|_{TV}\leq 0.05$ (for some $\Upsilon$ small enough), then this will enforce that the minimax separation distance $\rho^{*2}_{\gamma,\mathrm{var}}[0,\Delta]$ is larger than $c\Upsilon^{1/4}\sqrt{\Delta n^{1/2}}$.

Both $\mathbf{P}_{0}$ and $\mathbf{P}_{1}$ are product measures and can be decomposed as $\mathbf{P}_{0}=\pi_{0}^{\otimes n}$ and $\mathbf{P}_{1}=\pi_{1}^{\otimes n}$. By Cauchy Schwarz and by independence, we relate the total variation distance with the $\chi^{2}$ distance

[TABLE]

As a Consequence, it suffices to prove that $\int\frac{(d\pi_{1})^{2}}{d\pi_{0}}\leq 1+\frac{c}{n}$ for $c=\log(1+0.05^{2})$ to conclude that $\|\pi_{0}^{\otimes n}-\pi_{1}^{\otimes n}\|_{TV}\leq 0.05$. Expanding the integral, we get

[TABLE]

since $v^{2}=2pM^{2}$. Let $g_{1}$ and $g_{2}$ be the two functions defined by

[TABLE]

so that $\int\frac{(d\pi_{1})^{2}}{d\pi_{0}}=g_{1}(M^{2})g_{2}(M^{2})$. Observe that $g_{1}$ and $g_{2}$ are symmetric and infinitely differentiable on $(-1/p,1/p)$. Recall that $p\leq 1/4$. A fourth-order Taylor Lagrange inequality leads to

[TABLE]

where $c_{1}$ and $c_{2}$ are positive numerical constants constants. Since $M^{8}=\Upsilon\frac{n}{(\Delta^{\prime})^{2}}\leq 1$, we obtain that

[TABLE]

which is small enough if $\Upsilon$ is well-chosen. This concludes the proof.

Case 2 : $k_{0}>0$.

We follow the same lines as above except that we now take $\Delta^{\prime}=k_{0}+\Delta/2$. Since $\Delta\geq\sqrt{n}\geq k_{0}$, Bernstein’s inequality implies that $\mu_{1}^{\otimes n}[\|\theta\|_{0}\in[k_{0}+\Delta/4,k_{0}+\Delta]$ is close to one. Taking $\Upsilon$ small enough, we have $\|\mathbf{P}_{0}-\mathbf{P}_{1}\|_{TV}\leq 0.05$ as above. Thus, we conclude that

[TABLE]

since $\Delta\geq k_{0}$.

D.1.3 Proof of Proposition 5

We follow the same steps at in the proof of Theorem 3, except that we now fix $\Delta^{\prime}=n/3$ (and therefore $p=1/6$) and $M^{12}=\Upsilon/n$ with some $\Upsilon\in(0,1)$. Since $\Delta\geq n/3(1+\zeta)$ for some $\zeta>0$, Bernstein’s inequality enforces that $\mu_{1}^{\otimes n}[\|\theta\|_{0}\in[\Delta/2,\Delta]$ is close to one when $n$ is large enough. As a consequence, it suffices to prove that, for a suitable choice of $\Upsilon$, $\|\mathbf{P}_{0}-\mathbf{P}_{1}\|_{TV}$ is small enough to enforce that $\rho^{*2}_{\gamma,\mathrm{var}}[0,\Delta]$ is larger than $c\Upsilon^{1/6}n^{5/6}$. As in the previous proof, this amount to proving that $\int\frac{(d\pi_{1})^{2}}{d\pi_{0}}\leq 1+\frac{c^{\prime}}{n}$ for $c^{\prime}=\log(1+0.05^{2})$. As above this integral writes as $\int\frac{(d\pi_{1})^{2}}{d\pi_{0}}=g(M^{2})$ with

[TABLE]

In contrast to the general case, the choice $p=1/6$ has been precisely made to nullify the fourth-order expansion term of $g$. Since $g$ is symmetric and $g$ is infinitely differentiable is on $(-2,2)$, there exists a numerical constant $c>0$ such that $g(x)\leq 1+cx^{6}$ for all $x\in[-1,1]$, this implies that $\int\frac{(d\pi_{1})^{2}}{d\pi_{0}}\leq 1+\frac{c\Upsilon}{n}$. Taking $\Upsilon$ small enough concludes the proof.

D.1.4 Proof of Theorem 4

Without loss of generality, we assume that $\sigma_{+}=1$, $k_{0}\geq c\sqrt{n}$ ($c>0$ is a large enough universal constant) and that $k_{1}:=k_{0}+\Delta$ satisfies $n/2^{16}\geq k_{1}\geq 2^{16}k_{0}$. Set $\tilde{k}_{0}=k_{0}/2$, $\tilde{k}_{1}=k_{1}/2$, $p_{0}=\tilde{k}_{0}/n$ and $p_{1}=\tilde{k}_{1}/n$. Let $h_{0}$ and $h_{1}$ be two probability measures whose expression will be given later. We consider the probability measure

[TABLE]

and

[TABLE]

Obviously, $\mathbf{P}_{0}$ and $\mathbf{P}_{1}$ are product measures and decompose as $\mathbf{P}_{0}=\pi_{0}^{\otimes n}$ and $\mathbf{P}_{1}=\pi_{1}^{\otimes n}$. Note that $\pi_{0}$ is a convolution of the normal distribution with variance $1+\sigma_{0}^{2}$ with $\mu_{0}$ and $\pi_{0}$ is a convolution of the normal distribution with variance $1$ with the measure $\mu_{1}$.

By Chebychev’s inequality, we have

[TABLE]

for $n$ large enough. Also, the following lemma states that, with high probability, the vector $\theta$ sampled from $\mu_{1}^{\otimes n}$ is far from $\mathbb{B}_{0}[k_{0}]$.

Lemma 9**.**

For $h_{1}$ defined as in (129) below and for $n$ large enough, we have

[TABLE]

where $c$ is some positive universal constant.

Now consider any test $T$. Write $d_{n}=c\frac{\sqrt{k_{0}\Delta}}{\log(k_{0}/\sqrt{n})}$ where $c$ is the constant occurring in the above lemma. As in the proof of Theorem 1, we have

[TABLE]

As a consequence, the result of Theorem 4 holds as long as we are able to construct prior measures $h_{0}$ and $h_{1}$ such that $\|\pi_{0}^{\otimes n}-\pi_{1}^{\otimes n}\|_{TV}\leq 0.2$. By Cauchy-Schwarz inequality, we have

[TABLE]

As a consequence, it suffices to prove that

[TABLE]

Step 1 : Construction of the probability measures $h_{0}$ and $h_{1}$.

The purpose of this paragraph is to choose $h_{0}$ and $h_{1}$ in such a way that the characteristic function $\widehat{\pi}_{0}$ and $\widehat{\pi}_{1}$ of $\pi_{0}$ and $\pi_{1}$ match on the widest interval possible. Let us call $\widehat{h}_{0}$ and $\widehat{h}_{1}$ the characteristic function of $h_{0}$ and $h_{1}$.

It follows from the definition (123) of $\mu_{0}$ and $\mu_{1}$ that $\widehat{\mu}_{0}(t)=p_{0}\widehat{h}_{0}(t)+(1-p_{0})$ and $\widehat{\mu}_{1}(t)=(1-p_{1})+p_{1}\widehat{h}_{1}(t)$. Since $\pi_{0}$ (resp. $\pi_{1}$) are convolution production of $\mu_{0}$ (resp.$\mu_{1}$) with centered Gaussian measure with variance $(1+\sigma_{0}^{2})$ (resp. $\sigma_{1}$). We have

[TABLE]

To match $\widehat{\pi}_{0}(t)$ and $\widehat{\pi}_{1}(t)$, we therefore require that

[TABLE]

We start with some notation. Define $t^{*}=c^{*}\sqrt{\log(\tilde{k}_{0}^{2}/n)}$ with $c^{*}:=18$ and

[TABLE]

We first fix $\widehat{h}_{1}$ and then choose $\widehat{h}_{0}$ in such a way that (127) is satisfied on $[-t^{*},t^{*}]$.

[TABLE]

In other words, $h_{1}$ is a mixture of a Gaussian measure with variance $\sigma_{1}^{2}$ and of a Cauchy measure with parameter $\kappa$. For any $t\in[-t^{*},t^{*}]$, we define

[TABLE]

to satisfy (127). To conclude, it remains to prove that $\widehat{h}_{0}$ is the restriction to $[-t^{*},t^{*}]$ of the characteristic function of some probability measure (that will correspond to $h_{0}$).

Lemma 10**.**

If a symmetric function $g$ with $g(0)=1$ is convex and decreasing to [math] on $\mathbb{R}^{+}$, then $g$ is the characteristic function of some probability measure.

In view of the above lemma, it suffices to prove $\widehat{h}_{0}$ can be extended to satisfy the above property. The parameters $\sigma_{0}$, $\sigma_{1}$, $\kappa$ and $\lambda$ have been carefully chosen to ensure the following property.

Lemma 11**.**

Assume that $k_{1}\geq 6k_{0}$ and $k_{1}\leq n/14$. Then $\widehat{h}_{0}$ is positive, convex and decreasing on $[0,t^{*}]$.

For any $t\geq t^{*}$, we set $\widehat{h}_{0}(t)=(\widehat{h}_{0}(t^{*})+\widehat{h}_{0}^{\prime}(t^{*})(t-t^{*}))_{+}=:(a+b(t-t^{*}))_{+}$ , and for $t\in(-\infty,-t^{*})$ we simply take $\widehat{h}_{0}(t)=\widehat{h}_{0}(-t)$. In view of this extension, $\widehat{h}_{0}$ is continuous at $t^{*}$ and its slope is $\widehat{h}_{0}^{\prime}(t^{*})$. Hence, $\widehat{h}_{0}$ is a convex and decreasing on $\mathbb{R}^{+}$ and converges to 0 at $+\infty$. Since in addition $\widehat{h}_{0}$ is positive and $\widehat{h}_{0}(0)=1$, Lemma 10 ensures that $\widehat{h}_{0}$ is the characteristic function of a probability measures, denoted $h_{0}$ in the following.

Since $\widehat{h}_{0}$ is decreasing, positive and convex on $\mathbb{R}^{+}$, it follows that $a=\widehat{h}(t^{*})\in(0,1)$, and that $|b|\leq|\widehat{h}^{\prime}(0)|=(p_{1}/p_{0})(1-\lambda)\kappa\leq 1/(2t^{*})\leq 9$.

Step 2 : Upper bound of $A$ in terms of derivatives of Fourier transforms.

To simplify the notation, we write $\pi_{0}(x)$ (resp. $\pi_{1}(x)$) for the density corresponding to the probability measure $\pi_{0}$ and $\pi_{1}$. Recall that we aim (125) to upper bound

[TABLE]

Since $\pi_{1}$ is a mixture distribution with three components, one of which is a normal with variance $1$, we know that $\pi_{1}(x)\geq\tfrac{(1-p_{1})}{\sqrt{2\pi}}e^{-x^{2}/2}$, which implies since $p_{1}\leq 1/2$ that

[TABLE]

For any function defined on $\mathbb{R}$, denote $\|f\|_{2}$ its $l_{2}$ norm. Denote $P_{k}$ the polynom function $x\mapsto x^{k}$. Then, we take the Taylor expansion of the function $t\mapsto e^{t}$ to obtain

[TABLE]

by Plancherel formula and since $\widehat{x^{k}G}=i^{k}\widehat{G}^{(k)}/\sqrt{2\pi}$ (recall that $G$ is infinitely differentiable everywhere except at $-t^{*}$ and $t^{*}$).

Step 3 : Decomposition of $\|\widehat{G}^{(k)}\|_{2}^{2}$.

Our choice of $\widehat{\mu}_{0}$ and $\widehat{\mu}_{1}$ in Step 1 enforces that $\widehat{G}=\widehat{\pi}_{1}-\widehat{\pi}_{0}$ satisfies

[TABLE]

We have for any $t$ such that $|t|\geq t^{*}$

[TABLE]

For any $t$, let us write $V(t)=V_{1}(t)+V_{2}(t)+V_{3}(t)+V_{4}(t)$, where

[TABLE]

As a consequence, we have the decomposition

[TABLE]

which enforces

[TABLE]

We consider two subcases depending on the values of $k$: for small $k$ ($k\leq 5\log(\tilde{k}_{0}/\sqrt{n})$), we only need a loose upper bound of $(\phi V_{i})^{(k)}$ but we heavily rely on the fact that this derivative is null for $|t|\leq t^{*}$. For larger $k$, the computations need to be handled more carefully.

Step 4: Control of $\|\widehat{G}^{(k)}\|_{2}^{2}$ for $k\leq 5\log(\tilde{k}_{0}/\sqrt{n})$.

The binomial formula enforces that, for $i=1,\ldots,4$, $(\phi V_{i})^{(k)}=\sum_{d=0}^{k}\binom{k}{d}\phi^{(k-d)}V_{i}^{(d)}$, implying that

[TABLE]

Define $\overline{V}(t)=10p_{0}(t^{4}\vee 1)[e^{t/16}+e^{t^{2}/16}]$.

Lemma 12**.**

For all nonnegative integers $d$, all $t>0$ and all $i=1,\ldots,4$, one has

[TABLE]

Proof of Lemma 12.

Writing down the power expansion of $V_{1}$, we observe that the two first terms cancel out (recall that $p_{1}\lambda\sigma_{1}^{2}=\sigma_{0}^{2}$). As a consequence, the smallest order term is of order $p_{1}\lambda\sigma_{1}^{4}t^{4}\leq p_{0}t^{4}$. Besides all the terms of order $t^{2q+4}$ are smaller (in absolute value) than $(1/16)^{q}/q!$ because both $\sigma_{0}$ and $\sigma_{1}$ are small enough. This implies $V_{1}^{(d)}(t)\leq\overline{V}^{(d)}(t)$. The results for $V_{2}$, $V_{3}$ and $V_{4}$ follow similarly. ∎

Define the function $\phi_{+}:t\mapsto e^{t^{2}/2}/\sqrt{2\pi}$. For any nonnegative integer $k$, there exists a polynom $R_{k}$ of degree less or equal to $k$ such that $\phi^{(k)}_{+}(t)=R_{k}(t)\phi_{+}(t)$. By a straightforward induction on $k$, we observe that $|\phi^{(k)}(t)|\leq R_{k}(t)\phi(t)$. The same recursion allows us to prove that |\overline{V}^{(k)}(t)|\leq c\big{(}\max_{q=k,..,(k-3)_{+}}R_{q}(t)\big{)}|\overline{V}(t)|, where $c$ is a numerical constant. Also, we have $R_{k}R_{q}(t)\leq R_{k+q}(t)$. Coming back to (133), we obtain

[TABLE]

Write $R_{k}(t)=\sum_{j=0}^{k}r_{j,k}t^{j}$. Again, a straightforward induction leads to $0\leq r_{j,k}\leq\binom{k}{(k+j)/2}k^{(k-j)/2}\leq 2^{k}k^{(k-j)/2}$. Recall that $t^{*}\geq 1$. By the triangular inequality, we obtain

[TABLE]

Let us now bound this integral

[TABLE]

Coming back to (134), we obtain

[TABLE]

Thanks to (132), we conclude that, for $k\leq\log(\tilde{k}_{0}/\sqrt{n})=t^{*2}/(2c^{*2})$,

[TABLE]

where we used that $t^{*}=c^{*}\sqrt{\log(\tilde{k}^{2}_{0}/n)}$ with $c^{*}\geq 18$ and that $p_{0}=k_{0}/(2n)\geq cn^{-1/2}$ for a constant $c$ large enough.

Step 5: Control of $\|\widehat{G}^{(k)}\|_{2}^{2}$ for $k>5\log(\tilde{k}_{0}/\sqrt{n})$.

For such $k$, we may neglect the threshold $t\geq t^{*}$ but we need to be more careful about the computation of $(\phi V_{1})^{(k)}$.

[TABLE]

where $P_{k}:t\mapsto t^{k}$. Since $\phi V_{1}$ is a linear combination of normal distributions with different variances, we have

[TABLE]

A comparison of the power expansion ensures that, for any $x$, $|e^{x^{2}/2}-1-x|\leq\tfrac{x^{2}}{2}e^{x^{2}/2}$. Since $\sigma_{0}\leq\sigma_{1}$ and $p_{1}\lambda\sigma_{1}^{4}\leq p_{0}$, we obtain

[TABLE]

Coming back to (136), we conclude that

[TABLE]

Similarly, $\phi V_{2}$ is a difference of two normal distributions with different variances. Arguing as for $V_{1}$, we obtain

[TABLE]

Turning to $V_{3}$, we cannot directly apply (136) to the product $\phi V_{3}$. For $t\geq t*$, one has \phi V_{3}(t)=p_{1}(1-\lambda)\big{(}e^{\kappa^{2}/2}e^{-(t-\kappa)^{2}/2}-e^{-t^{2}/2}\big{)}. Let $W$ be the function defined on $\mathbb{R}$ by this last expression.

[TABLE]

Let us compute the Fourier transform of $W$.

[TABLE]

Hence, we conclude that

[TABLE]

Finally, we consider $V_{4}$. Observe that

[TABLE]

We then conclude

[TABLE]

Gathering (137), (138), (139), and (140), we get

[TABLE]

since $\sigma_{0}^{2}\leq 0.1$ and $k\geq 5\log(\sqrt{n}p_{0}/2)$ and $p_{0}$ is small enough.

Step 6: Conclusion

Coming back to (131), we obtain by (135) and (141) that

[TABLE]

We have proved inequality (125) and this concludes the proof.

Proof of Lemma 9.

We write $N^{\theta}_{\sigma_{1}}$ for the number of coordinates of $\theta$ that are larger than $\sigma_{1}$ in absolute value. One of the components of the mixture distribution $h_{1}$ is a centered normal distribution with variance $\sigma_{1}^{2}$ and has weight $\lambda_{1}\tilde{k}_{1}/n\geq k_{1}/4n$. Hence, $N^{\theta}_{\sigma_{1}}$ is stochastically larger than a Binomial distribution with parameter $(n,k_{1}/(8n))$. By Chebychev’s inequality, we have $\mu_{1}^{\otimes n}[N^{\theta}_{\sigma_{1}}<k_{1}/16]\leq 0.1$ for $n$ large enough. Since we assumed at the beginning of the proof of Theorem 4 that $k_{1}\geq 32k_{0}$, this implies that, with probability larger than $0.9$,

[TABLE]

by definition (128) of $\sigma_{1}$. ∎

Proof of Lemma 11.

Note first that by definition (128) of the parameters the following conditions hold :

[TABLE]

Step 1 : positivity of $\widehat{h}_{0}$ on $[0,t^{*}]$. We first check that $\widehat{h}_{0}(t)\geq 0$ for any $t\in[-t^{*},t^{*}]$, ie.  we check that e^{\sigma_{0}^{2}t^{2}/2}\big{[}1-p_{1}+p_{1}\widehat{h}_{1}(t)\big{]}\geq 1-p_{0}. By definition (128), we have, for $t\in[0,t^{*}]$, that $\sigma_{0}^{2}t^{2}/2\leq 1/2$ and that $\sigma_{1}^{2}t^{2}/2\leq 1/2$ and $\kappa|t|\leq 1/2$. We get

[TABLE]

Relying on (142) and (128), we have, for $t\in[0,t^{*}]$,

[TABLE]

which is positive since $p_{0}=k_{0}/(2n)$ is small enough.

Step 2 : negativity of $\widehat{h}^{\prime}_{0}(t)$ on $[0,t^{*}]$. We have

[TABLE]

so that, for any $t\in[0,t^{*}]$, we have

[TABLE]

where we used again (142) and the definition (128) of $\sigma_{0}$ and $\sigma_{1}$. Since $p_{0}/p_{1}\leq 1/2$, this last expression is nonpositive.

Step 3 : positivity of $\widehat{h}^{\prime\prime}_{0}(t)$ on $[0,t^{*}]$. Deriving two times $\widehat{h}_{0}$, we get

[TABLE]

Let us bound $\widehat{h}_{1}$ and its derivatives for $t\in[0,t^{*}]$. Since, for $x\geq 0$, we have $1\geq e^{-x}\geq 1-x$, it follows that

[TABLE]

since $\sigma_{1}^{2}t^{*2}\leq 1/4$ and $\kappa t^{*}\leq 1/2$. Gathering these bounds, we get

[TABLE]

since $6\sigma_{0}^{2}\leq\sigma_{1}^{2}$ by (142) and as we may suppose that $p_{1}=(k_{0}+\Delta/2)/n\leq 1/6$. By (128) $\kappa=4\sigma_{0}^{2}t^{*}/\lambda\geq 4\sigma_{0}^{2}t^{*}$ and $\kappa t^{*}=0.5\sqrt{p_{0}/p_{1}}$ that we may suppose to be smaller than $1/4$. We have proved that $\widehat{h}^{\prime\prime}_{0}(t)\geq 0$ for all $t\in[0,t^{*}]$.

∎

D.2 Proofs of the upper bounds

By homogeneity, we assume henceforth that $\sigma_{+}=1$, which is equivalent to considering $Y^{\prime}=Y/\sigma_{+}$ whose noise variance belongs to $[\sigma^{2}_{-}/\sigma^{2}_{+},1]$.

D.2.1 Proof of Theorem 5

For the sake of simplicity, we denote $t_{*}$ for $t_{*,\alpha}^{HC,\mathrm{var}}$. For any integer $q$ and any $x>0$, we denote

[TABLE]

Let us write $\widehat{\sigma}^{2}$ for $\widehat{\sigma}^{2}(v)$ in order to simplify notation. Also denote

[TABLE]

The key step of this proof is to control the difference between the tail probabilities $\Phi[t/\sigma]$ and the estimated probabilities $\Phi[t/\widehat{\sigma}]$. To do this, we shall rely on the two following lemmas. The first one control the estimation error of $\sigma$ whereas the second one quantifies the error propagation for the tail probabilities.

Lemma 13**.**

Consider any vector $\theta$ satisfying $48\|\theta\|_{0}\leq n$. For any $x>0$, the estimator $\widehat{\sigma}^{2}$ satisfies

[TABLE]

with probability larger than $1-2e^{-x}$.

Lemma 14**.**

Let $a>0$ and $b>0$ be such that $-a\leq\widehat{\sigma}^{2}-\sigma^{2}\leq b$. We have

[TABLE]

Level of the test. Consider any $\theta\in\mathbb{B}_{0}[k_{0}]$. For any $t>0$, $N_{t}-k_{0}$ is stochastically bounded by a Binomial distribution with parameters $n-k_{0}$ and 2\Phi\big{[}t/\sigma\big{]}. Since $\Phi(t/\sigma)\leq\exp(-t^{2}/(2\sigma^{2}))$, we obtain by a simple union bound that since $\sigma\leq\sigma^{+}=1$

[TABLE]

For the statistic $N_{t}$ with smaller $t$, we first need to control the estimated variance $\widehat{\sigma}^{2}$. By Lemma 13, we have

[TABLE]

on an event of probability larger than $1-\alpha/3$. We assume henceforth that this event is true. In view of the definition (144) of $d_{\sigma}^{-}$, we have $d_{\sigma}^{-}\sqrt{\log(6/\alpha)}\leq\sigma^{2}/2$ when $n$ is large in front of $\alpha$.

Consider any $t\in\mathbb{N}^{*}$. Bernstein’s inequality ensures that

[TABLE]

outside an event of probability smaller than $(2\alpha)/(\pi^{2}t^{2})$. Gathering the bound (148) together with Lemma 14, we deduce that

[TABLE]

and therefore

[TABLE]

Taking an union bound over all $t\in\mathbb{N}^{*}$, we conclude that the type I error probability of $T^{HC,\mathrm{var}}_{\alpha,k_{0}}$ is smaller or equal to $\alpha$.

Power of the test. Consider any $\theta$ satisfying Condition (52) and fix $q\in[n-k_{0}]$. We will prove that, if $|\theta_{(k_{0}+q)}|$ is large enough so that Condition (51) is satisfied, the type II error probability of the test is smaller than $\beta$. We consider separately small and large values of $q$. Define q_{+}:=L_{\alpha,\beta}\big{[}1+\log(C)+\log(\frac{1}{\sigma_{-}}\big{)}+\log\big{(}k_{0}\vee\sqrt{n})\big{]}, where the constant $L_{\alpha,\beta}$ will be fixed at the end of the proof.

Case 1: $q\leq q_{+}$. We focus on the statistic $N_{t_{*}}$ and we have :

[TABLE]

Restricting ourselves to the $k_{0}+1$ largest values of $\theta$, we get

[TABLE]

Since $\Phi(x)\leq e^{-x^{2}/2}$ for any $x>0$,the type II error probability is smaller than $\beta$ as soon as $\theta_{(k_{0}+q)}\geq t_{*}+\sigma\sqrt{2\log((k_{0}+1)/\beta)}$. For $q\leq q_{+}$, this condition is ensured by (51).

Case 2: $q>q_{+}$. Recall that for $t>0$, $N_{t}^{\theta}$ refers to the number of components of $\theta$ larger or equal to $t$ (in absolute value). Let $t$ be a positive integer larger than $4\sigma_{+}$ whose value will be fixed later (see (154) below) . We shall prove that, as long as $|\theta_{(k_{0}+q)}|\geq 2t$, the statistic $N_{t}$ takes large values so that the type II error probability of the test is smaller than $\beta$.

Let us first control the difference $\Phi(\frac{t}{\widehat{\sigma}})-\Phi(\frac{t}{\sigma})$ using Lemmas 13 and 14. Since Condition (52) ensures that $N_{1/v}^{\theta}\leq C(k_{0}\vee\sqrt{n})$ and $v^{2}=2\log(1+\frac{k_{0}}{\sqrt{n}})\vee 1$, it follows from the definition of $d_{\sigma}^{+}$ that

[TABLE]

By Cauchy-Schwarz inequality and Condition (52), $(M^{\theta}_{2,1/v})^{2}\leq nM_{4,1/v}^{\theta}\leq nCv^{-4}(k_{0}\vee\sqrt{n})$. As a consequence, we have

[TABLE]

for $n$ large enough in front of $\beta$, $\sigma_{-}$ and $C$. Besides, Lemma 13 ensures that \widehat{\sigma}-\sigma^{2}\leq d_{\sigma}^{+}+d_{\sigma}^{-}\sqrt{\log\big{(}\frac{4}{\beta}\big{)}} with probability larger than $1-\beta/2$. Under this event, we have, by Lemma 14, that

[TABLE]

To control $N_{t}$, we divide the components of $\theta$ into three groups: the $(k_{0}+q)$ largest (in absolute value) components of $\theta$ which are, by assumption, all larger than than $2t$, those smaller than $1/v$, and the remaining components. Thus, the statistic $N_{t}$ is stochastically lower bounded the random variable $S$, where $S$ is the sum of a Binomial random variable with parameters ($k_{0}+q,1-\Phi(t/\sigma))$, a Binomial random variables with parameters $(N^{\theta}_{1/v}-(k_{0}+q),2\Phi(t/\sigma))$, and $\sum_{i:|\theta_{i}|<1/v}\mathbf{1}_{|Y_{i}|\geq t}$. By Bernstein’s inequality, we have

[TABLE]

We first control $\operatorname{\mathbb{E}}[S]$. In the definition of $S$, only the expectation of $\sum_{i:|\theta_{i}|<1/v}\mathbf{1}_{|Y_{i}|\geq t}$ is difficult to handle.

Lemma 15**.**

For any $t\geq 4\sigma$ and $0\leq x\leq t/2$, it holds that

[TABLE]

Since $t\geq 4\geq 2/v$, it follows from Lemma 15 above that

[TABLE]

Together with (150), we get

[TABLE]

Since the variance of a Binomial random variable with parameters $n$ and $p$ is upper bound by $n(p\wedge(1-p))$, we derive from Lemma 15 that

[TABLE]

where we used again Condition (52) in the second line. In view of (49), (151), (152), and (153), the type II error probability of $T^{HC,\beta}_{\alpha,k_{0}}$ is smaller than $\beta$ as soon

[TABLE]

Since $\Phi(x)\leq e^{-x^{2}/2}$, we $A_{1}\leq q/2$ as soon as we choose

[TABLE]

for some constant $c^{\prime}_{\alpha,\beta}$ large enough. Fixing $t=\lceil t_{q}^{0,\mathrm{var}}\rceil$, we also have $A_{2}\leq q/2$ for $q>q_{+}$ which we can assume if we take the constant $L_{\alpha,\beta}$ large enough in the definition of $q_{+}$.

Proof of Lemma 13.

Since the cosinus function if bounded, Hoeffding’s inequality ensures that

[TABLE]

for any $x>0$. Recall that the characteristic function writes as $\overline{\varphi}(v)=e^{-v^{2}\sigma^{2}/2}\sum_{i=1}^{n}\frac{\cos(v\theta_{i})}{n}$. Using the Taylor expansion of $\cos(x)$, we derive that $1\geq\cos(x)\geq 1-x^{2}/2+x^{4}/48$ for any $x\in(-1,1)$. Considering separately the components $v\theta_{i}$ that are smaller or larger than one (in absolute value), we get

[TABLE]

Since $v^{2}M^{\theta}_{2,1/v}\leq\|\theta\|_{0}$ and $N^{\theta}_{1/v}\leq\|\theta\|_{0}$, the condition $48\|\theta\|_{0}\leq n$ implies that the expression in lhs is larger than $1/2$. For $x\in(0,1/2)$, $\log(1-x)\geq-x-2x^{2}$, implying that

[TABLE]

where we used Cauchy-Schwarz inequality and $4N_{1/v}^{\theta}\leq 4\|\theta\|_{0}\leq n$ in the third line and $48\|\theta\|_{0}\leq n$ in the last line. It follows from (155) that, with probability larger than $1-2x$,

[TABLE]

since $\sum_{i=1}^{n}\cos(v\theta_{i})\geq n/2$. Together with (156), we conclude that

[TABLE]

Since 2v^{-2}e^{v^{2}\sigma^{2}/2}\leq e\frac{k_{0}}{n\log\big{(}1+\frac{k_{0}}{\sqrt{n}}\big{)}}, we have proved (145).

∎

Proof of lemma 14.

fix $t>0$ and denote \delta_{t}:=\frac{t}{\widehat{\sigma}}-\frac{t}{\sigma}=\frac{t}{\sigma}\big{[}(1+\frac{\widehat{\sigma}^{2}-\sigma^{2}}{\sigma^{2}})^{-1/2}-1]. Applying the Taylor formula to the function $x\mapsto\Phi(x)$, we get

[TABLE]

if $\delta_{t}<0$, whereas this difference is bounded by [math] when $\delta_{t}\geq 0$. We now need to bound $\delta_{t}$ in terms $\widehat{\sigma}-\sigma$. By convexity, we have $(1+x)^{-1/2}\geq 1-x/2$ for any $x>-1$. It then follows that $\delta_{t}\geq-\frac{tb}{2\sigma^{3}}$.

[TABLE]

Turning to the lower bound (146), we use the convexity of the function $x\mapsto\Phi(x)$ in the second line.

[TABLE]

For any $x\in[-1/2,0]$, we have $(1+x)^{-1/2}\leq 1-x$. Taking $x=\min(\tfrac{\widehat{\sigma}^{2}-\sigma^{2}}{\sigma^{2}},0)\geq-\frac{a}{\sigma^{2}}\geq-1/2$, we obtain

[TABLE]

∎

Proof of Lemma 15.

Fix $t\geq 4\sigma$ and consider the function

[TABLE]

It holds that $h^{\prime}(0)=0$ and

[TABLE]

Next, we show that $h^{\prime\prime}$ is increasing on $[0,t/2]$. We have

[TABLE]

where $k:x\mapsto(x-1)e^{-x/2}$. Observe that the function $k$ is decreasing on $[3,\infty]$. For $x\leq t/2$ and $t\geq 4\sigma$, $(t-x)/\sigma\geq\sqrt{3}$ and $h^{\prime\prime\prime}(x)$ is therefore positive. Relying on $h(0)=h^{\prime}(0)=0$ as well as $h^{\prime\prime}(t/2)\geq h^{\prime\prime}(u)\geq h^{\prime\prime}(0)$ for any $u\in[0,x]$, we obtain by Taylor’s theorem that

[TABLE]

This concludes the proof of the lemma. ∎

D.2.2 Proof of Theorem 6

For the sake of simplicity, we simply write $s$ for $s_{k_{0}}^{\mathrm{var}}$ in this section. In order for the statistic $Z^{\mathrm{var}}(s)$ to be properly defined, the process $\overline{\varphi}_{n}(s\xi)>0$ for $\xi\in[0,1]$ has to be positive. This will turn out to be true when the following event holds.

[TABLE]

holds.

Lemma 16**.**

For any $a>1$ and any $\theta\in\mathbb{R}^{n}$, we have

[TABLE]

As a consequence \operatorname{\mathbb{P}}_{\theta}(\mathcal{A}^{c})\leq e^{-n/2}+\frac{1}{n^{3}}\big{(}1+\frac{\|\theta\|_{1}}{n}\big{)}.

The following proposition characterizes the deviations of the statistic $Z^{\mathrm{var}}(s)$. Denote $N_{1/s}^{\theta}:=|\{i:\ |\theta_{i}|>s^{-1}\}|$ the number of coordinates larger than $1/s$.

Proposition 10**.**

There exist numerical constants $c_{1}$, $c_{2}$, $c_{3}$ and $c_{4}$ such that the following holds. Assume that $n\geq c_{1}$, $\|\theta\|_{0}\leq n/c_{2}$. For any $x\geq 2$, the statistic $Z^{\mathrm{var}}(s)$ satisfies

[TABLE]

on the intersection of $\mathcal{A}$ and an event of probability larger than $1-2e^{-x}$.

Theorem 6 is a straightforward consequence of Proposition 10 and Lemma 16. The first upper bound in the above proposition ensures that the type I error is smaller than $\alpha+\mathcal{P}_{\theta}[\mathcal{A}^{c}]$. With probability larger than $1-\beta-\operatorname{\mathbb{P}}_{\theta}[\mathcal{A}^{c}]$, the statistic $Z^{\mathrm{var}}(s)$ is larger than

[TABLE]

and the test rejects the null hypothesis as soon as this expression is larger than

[TABLE]

This is the case if either $N_{1/s}^{\theta}$ or $\sum_{i=1}^{n}(s\theta_{i})^{4}\mathbf{1}_{|s\theta_{i}|\leq 1}$ is large enough, which is precisely ensured by Condition (57).

Proof of Proposition 10.

Assume that $\|\theta\|_{0}\leq n/40$. Under the event $\mathcal{A}$ (defined in (158)), the empirical characteristic function satisfies

[TABLE]

for $n$ large enough since $s^{2}\sigma^{2}\leq 1+\log(n)/2$. As a consequence, the empirical characteristic functions $\overline{\varphi}_{n}(u)$ is positive on $[0,s]$ and the statistic $Z^{\mathrm{var}}(s)$ is properly defined.

By definition of $P_{B}$, $\int_{0}^{1}P_{B}(\xi)\xi^{2}d\xi=0$. Hence,

[TABLE]

To control the behavior of the statistic, we linearize the logarithm. For any $x\in[0.9,1.1]$, it holds that $|\log(1+x)-x|\leq 2x^{2}/3$. Hence, under the event $\mathcal{A}$, the statistic $Z^{\mathrm{var}}(s)$ satisfies

[TABLE]

In the above bound, we decompose the deterministic and random quantities as follows

[TABLE]

so that

[TABLE]

In the remainder of the proof, we control each of these four quantities.

Control of $A_{1,1}$. Relying on the definition of $P_{B}(\xi)=4\xi-3$, we explicitly compute the trigonometric integral

[TABLE]

Define the symmetric function $g$ by $g(0)=0$ and $g(x):=1+\frac{\sin(x)}{x}+4\frac{\cos(x)-1}{x^{2}}$ for $x\neq 0$.

Lemma 17**.**

The function $g$ is supported in $[0,1.09)$ and satisfies

[TABLE]

Hence, we conclude that

[TABLE]

Control of $A_{2,1}$. In this second order deterministic term, we also separately handle small and large coordinates of $\theta$. For any $\xi\in[0,1]$, it holds that

[TABLE]

Since $\int_{0}^{1}|P_{B}(\xi)|d\xi\leq 2$ and $\int_{0}^{1}|P_{B}(\xi)|\xi^{4}d\xi\leq 3$, we arrive at

[TABLE]

Simply bounding |\sum_{i=1}^{n}\cos\big{(}s\theta_{i}\xi\big{)}-1| by $2\|\theta\|_{0}$, we also have $A_{2,1}\leq 8[\|\theta\|_{0}/n]^{2}$. Together with (162), this yields

[TABLE]

Turning to a lower bound of $A_{1,1}-2A_{2,1}$, we observe that the expressions in (162) and (163) counterbalance

[TABLE]

assuming that $\|\theta\|_{0}/n\leq\tfrac{g(1)}{16}\wedge\tfrac{2}{7!}$.

Control of $A_{1,2}$. Let $X\sim\mathcal{N}(x,\sigma^{2})$. The random variable $\int_{0}^{1}P_{B}(\xi)e^{s^{2}\xi^{2}\sigma^{2}/2}\cos(s\xi X)d\xi$ is smaller in absolute value than $e^{s^{2}\sigma^{2}/2}\int_{0}^{1}|P_{B}(\xi)|d\xi\leq 2e^{s^{2}\sigma^{2}/2}$. Hence, Hoeffding’s inequality yields

[TABLE]

for any $x>0$.

Control of $A_{2,2}$. The event $\mathcal{A}$ ensures uniform bound on the difference $\overline{\varphi}_{n}(u)-\overline{\varphi}(u)$. As a consequence,

[TABLE]

Since $s^{2}\sigma^{2}\leq 1+\log(n)/2$, this term is small in front of the first order term $A_{1,2}$ for $n$ large enough, that is $|A_{2,2}|\leq e^{s^{2}\sigma^{2}/2}/\sqrt{n}$. We conclude that, for any $x\geq 1$, $|A_{1,2}|+2|A_{2,2}|$ is smaller than $4e^{s^{2}\sigma^{2}/2}\sqrt{\tfrac{x}{n}}$ on the intersection of $\mathcal{A}$ and an event of probability larger than $1-2e^{-x}$. Together with (160), (165) and (164), this concludes the proof.

∎

Proof of Lemma 16.

Denote $u_{*}:=\sqrt{2\log(n)}$. Let $K$ be an integer whose value will be fixed later. By Hoeffding’s inequality, we have, for any $u>0$ and $x>0$,

[TABLE]

Fix $x>0$. Applying an union bound, we obtain, that, with probability larger than $1-2Ke^{-x}$,

[TABLE]

Since the function $x\mapsto\cos(x)$ is $1$-Lipschitz, we have

[TABLE]

for any $u\neq u^{\prime}$. By the Gaussian concentration theorem, we have $\sum_{i=1}^{n}|\epsilon_{i}|\leq 2\sigma n\leq 2n$ with probability larger than $1-\exp(-n/2)$. Taking the expectation in the above inequality also leads to

[TABLE]

For any $u\in[0,u_{*}]$, there exists $j$ such that $|u-ju_{*}/K|\leq u_{*}/K$. With probability larger than $1-2Ke^{-x}-e^{-n/2}$, we therefore have

[TABLE]

Setting $K=n[1+\frac{\|\theta\|_{1}}{n}]$ and $x=a\log(n)$ for any $a>1$ yields the first result. Then, fixing $a=4$ yields the second result.

∎

Proof of Lemma 17.

Fist we consider the behavior of $g(x)$ for $|x|\geq 2\pi$. Since $\cos^{2}(x)+\sin^{2}(x)=1$,

[TABLE]

As a consequence, $g(x)\geq 0.7$ for $|x|>2\pi$. Besides, studying the behavior of the function $(-4+\sqrt{x^{2}+16})/x^{2}$ for $|x|\geq 2\pi$, we also conclude that $g(x)\leq 1.09$ for $|x|\geq 2\pi$.

Then, we prove that $g$ is non-decreasing on $[0,2\pi]$. To do this, we study the sign of $h(x):=x^{3}g^{\prime}(x)=x^{2}\cos(x)-5x\sin(x)+8(1-\cos(x))$. Since $h^{\prime\prime}(x)=x[\sin(x)-x\cos(x)]$, we observe by considering the sign of the derivative of $(h^{\prime\prime}(x)/x)$ that $h^{\prime\prime}(x)$ is first increasing from $h^{\prime\prime}(0)=0$ and then decreasing to $h^{\prime\prime}(2\pi)<0$. Thus, $h^{\prime}(x)$ is therefore also increasing from $h^{\prime}(0)$ and then decreasing to $h^{\prime}(2\pi)<0$. Since $h(0)=0$ and $h(2\pi)>0$, this implies that $g$ is increasing on $[0,2\pi]$.

As consequence of the two above results, we conclude that $\inf_{x>1}g(x)\geq g(1)\wedge 0.7=g(1)$.

For $|x|$ smaller than $1$, we come back to the definition of $g(x)=\int_{0}^{1}P_{B}(t)[\cos(tx)-1]dt$. By Taylor’s inequality, we get $|\cos(tx)-1+(t^{2}x^{2})/2+(t^{4}x^{4})/4!|\leq t^{6}x^{6}/6!$. Together with the identity $\int_{0}^{1}P(t)t^{2}dt=0$, this yields

[TABLE]

which allows us to conclude since $x^{6}\leq x^{4}$.

∎

D.2.3 Proof of Theorem 7

For the sake of clarity with $r_{l}$ for $r_{k_{0},l}$ in the remainder. First observe that for all $l\in\mathcal{L}_{k_{0}}$, $r_{l}\geq 4$ which implies

[TABLE]

The following proposition characterizes the deviations of the statistics $V^{\mathrm{var}}(r_{l},w_{l})$.

Proposition 11**.**

There exist two positive constants $c$ and $c$ such that the following hold. Assume that $n\geq c$ and consider any vector $\theta$ satisfying $\|\theta\|_{0}\leq c^{\prime}n$. For any $x\geq 1$ and any $l\in\mathcal{L}_{k_{0}}$, the statistic $V^{\mathrm{var}}(r_{l},w_{l})$ satisfies

[TABLE]

on the intersection of the event $\mathcal{A}$ (defined in (158)) and an event of probability larger than $1-2e^{-x}$.

We first prove how Theorem 7 derives from the above proposition.

Proof of Theorem 7.

The control of the type I error probability is a straightforward consequence of Lemma 16 and Proposition 11 together with an union bound over all $l\in\mathcal{L}_{k_{0}}$ with weights $\tfrac{3\alpha}{\pi^{2}}[1+\log_{2}(l/l_{0})]^{-2}$.

Let us turn to the type II error. Denote $a_{0}:=1/s_{k_{0}}^{\mathrm{var}}$, where we recall that $s_{k_{0}}^{\mathrm{var}}=\sqrt{\log(ek_{0}/\sqrt{n})}$. For all $l\in\mathcal{L}_{k_{0}}$, we have $l\leq k_{0}$ implying that $N_{1/w_{l}}^{\theta}\leq N_{a_{0}}^{\theta}$. Consider any parameter $\theta$ satisfying $N_{a_{0}}^{\theta}\leq Ck_{0}$ for some $C>2$. For any $l\in\mathcal{L}_{k_{0}}$, Proposition 11 ensures that

[TABLE]

with probability larger $1-\beta-\mathcal{P}_{\theta}[\mathcal{A}^{c}]$. Since $e^{s^{2}_{l}\sigma^{2}/2}\leq l^{1/2}n^{-1/4}$, it follows from the definition (61) of $T^{I,\mathrm{var}}_{\alpha,k_{0}}$ that the type II error probability smaller than $\beta+\operatorname{\mathbb{P}}_{\theta}[A^{c}]$, if there exists $l\in\mathcal{L}_{k_{0}}$ such that

[TABLE]

Since $l\geq l_{0}\geq\sqrt{n^{1/2}k_{0}}$, the last expression in the rhs is smaller than $c_{\alpha,\beta}l$. By definition (60) of $\delta_{l}$ and since $r_{k}\geq 4$, it holds that $\delta_{l}k_{0}\leq 4r_{l}\phi(r_{l})k_{0}\leq 16l(\tfrac{l}{k_{0}})^{7}\sqrt{\log(k_{0}/l)}$. As a consequence, Condition (168) simplifies as

[TABLE]

which is equivalent to

[TABLE]

To conclude, it suffices to prove that, with suitable constants, Condition (63) enforces (169). Assume that $\theta$ satisfies Condition (63) for some $q$. Define $l(q):=\max\{l\in\mathcal{L}_{k_{0}},\ \text{such that}\,q\geq 2c_{\alpha,\beta}Cl\}$. Since $q$ is large in front of $C\sqrt{k_{0}n^{1/2}}$, this implies that $l(q)\geq l_{0}$. As a consequence, for some constant $c^{\prime\prime}_{\alpha,\beta}$, it holds that

[TABLE]

and therefore

[TABLE]

If the constant $c^{\prime\prime}_{\alpha,\beta}$ in (63) is set to $\overline{c}_{\alpha,\beta}$, then $N_{r_{l}^{2}/w_{l}}^{\theta}\geq k_{0}+q$, which implies that (169) is satisfied for $l=l(q)$. Thus, the type II error probability is smaller than $\beta+\operatorname{\mathbb{P}}_{\theta}[\mathcal{A}^{c}]$. ∎

Proof of Proposition 11.

Assume that $\|\theta\|_{0}\leq n/40$. Observe that for all $l\in\mathcal{L}_{k_{0}}$, $w_{l}^{2}\leq\log(n)/2$. Under the event $\mathcal{A}$ (defined in (158)), the empirical characteristic function satisfies

[TABLE]

for $n$ large enough. As a consequence, the empirical characteristic function $\overline{\varphi}_{n}(u)$ is positive on $[0,w_{l}]$ for $l\in\mathcal{L}_{k_{0}}$ and the statistics $V^{\mathrm{var}}(r_{l},w_{l})$ are properly defined.

Fix some $l\in\mathcal{L}_{k_{0}}$. As the polynomial $P_{l}$ has been chosen in such a way that $\int_{-r_{l}}^{r_{l}}P_{l}(\xi)\phi(\xi)\xi^{2}d\xi=0$, we have

[TABLE]

As for the statistic $Z^{\mathrm{var}}(s)$, we then linearize the logarithm. For any $t\in[0.9,1.1]$, $|\log(1+t)-t|\leq 2t^{2}/3$. Hence, under the event $\mathcal{A}$, $V^{\mathrm{var}}(r_{l},w_{l})$ satisfies

[TABLE]

In the above bound, we decompose the deterministic and random quantities as follows

[TABLE]

so that

[TABLE]

In the sequel, we control these four quantities.

Control of $A_{1,1}$. We first focus on the deterministic quantity $A_{1,1}$. Define the function $\Psi^{\mathrm{var}}_{l}$ by

[TABLE]

so that $A_{1,1}=n^{-1}\sum_{i=1}^{n}[\Psi^{\mathrm{var}}_{l}(\theta_{i})-\Psi^{\mathrm{var}}_{l}(0)]$. The following lemma provides bounds for function $\Psi^{\mathrm{var}}_{l}$.

Lemma 18**.**

The function $\Psi^{\mathrm{var}}_{l}$ satisfies

[TABLE]

for any $x\in\mathbb{R}$. This implies that

[TABLE]

Finally, for all $x\in[-1/w_{l};1/w_{l}]$,

[TABLE]

Recall that $\gamma_{l}\geq 1/3$ by (167). As a consequence, we obtain the following the bound for $A_{1,1}$

[TABLE]

Control of $A_{2,1}$. As for $A_{1,1}$ we consider separately the coordinates larger than $1/w_{l}$ and the coordinates smaller than $1/w_{l}$.

[TABLE]

Relying on the bounds (167) for $\zeta_{l}$, $\gamma_{l}$ and $\kappa_{l}$, we derive that $\int_{-r_{l}}^{r_{l}}|P_{l}(\xi)|\phi(\xi)d\xi\leq\gamma_{l}\int_{\mathbb{R}}(\zeta_{l}\xi^{2}+\kappa_{l})\phi(\xi)d\xi\leq 4$ and $\int_{-r_{l}}^{r_{l}}|P_{l}(\xi)|\xi^{4}\phi(\xi)d\xi\leq\gamma_{l}\int_{\mathbb{R}}(\zeta_{l}\xi^{6}+\kappa_{l}\xi^{4})\phi(\xi)d\xi\leq(15\zeta_{l}+4\kappa_{l})\gamma_{l}\leq 27$ , we arrive at

[TABLE]

In the first line of the above derivation, we may also simply bound |\sum_{i=1}^{n}\cos\big{(}\frac{w_{l}}{r_{l}}\theta_{i}\xi\big{)}-1| by $2\|\theta\|_{0}$ to obtain $A_{2,1}\leq 16\|\theta\|_{0}^{2}/n^{2}$. Together with (175), this yields

[TABLE]

Turning to the lower bound of $A_{1,1}-2A_{2,1}$, we observe that the terms in $\theta_{i}^{4}$ in (177) counterbalanced by those in (176)

[TABLE]

assuming that $\|\theta\|_{0}/n$ is small enough.

Control of $A_{1,2}$. Let $X\sim\mathcal{N}(x,\sigma^{2})$. The random variable \int_{-r_{l}}^{r_{l}}P_{l}(\xi)\phi(\xi)\exp\big{(}\frac{w_{l}^{2}\sigma^{2}\xi^{2}}{2r_{l}^{2}}\big{)}\cos(\frac{w_{l}}{r_{l}}\xi X)d\xi is smaller in absolute value than $e^{w_{l}^{2}\sigma^{2}/2}\int_{\mathbb{R}}|P_{l}(\xi)|\phi(\xi)d\xi\leq 4e^{w_{l}^{2}\sigma^{2}/2}$. As a consequence, Hoeffding’s inequality yields

[TABLE]

for any $x>0$.

Control of $A_{2,2}$. We use the event $\mathcal{A}$ (Eq.(158)), to uniformly bound the difference $\overline{\varphi}_{n}(u)-\overline{\varphi}(u)$.

[TABLE]

Since $w_{l}^{2}\sigma^{2}\leq\log(n)/2$, this term is negligible is small in front of the first order term $A_{1,2}$ for $n$ large enough, that is

[TABLE]

We conclude that, for any $x\geq 1$, $|A_{1,2}|+2|A_{2,2}|$ is smaller than $8e^{w_{l}^{2}\sigma^{2}/2}\sqrt{x/n}$ on the intersection of $\mathcal{A}$ and an event of probability larger than $1-2e^{-x}$. Together with (170), (178) and (179), this concludes the proof.

∎

Proof of Lemma 18.

For the sake of simplicity, we simply write $r$, $w$, $\gamma$, and $\delta$ for $r_{l}$, $w_{l}$, $\gamma_{l}$, and $\delta_{l}$ in the remainder of this proof. As for the function $\Psi_{l}$ corresponding to the statistic with known variance, we decompose the integral in $\Psi^{\mathrm{var}}_{l}(x)$ to obtain the Fourier transform of a standard normal distribution

[TABLE]

where we used the integration by part in the second line. Let us now upper bound the second expression in the rhs.

[TABLE]

where we used again the integration by part and (167). Gathering the two above inequalities yields

[TABLE]

We have proved (172). Consider the function $h:u\mapsto\sqrt{2\pi}\phi(u)[1+\frac{\zeta}{\kappa-\zeta}u^{2}]$ defined on $\mathbb{R}^{+}$. Studying the sign of its derivative, we observe that it is maximized at $u_{*}^{2}=\frac{3\zeta-\kappa}{\zeta}=\frac{2r^{3}\phi(r)}{\zeta}\leq 1/2$ since $r\geq 4$. As a consequence of (167), we obtain

[TABLE]

where $4r^{6}\phi^{2}(r)\leq\delta$ since $r\geq 4$. Plugging this bound into (172) yields the first part of (173). For $x\geq r^{2}/w$, we have, since $r\geq 4$,

[TABLE]

implying the second part of (173).

It remains to control $\Psi^{\mathrm{var}}_{l}(x)-\Psi^{\mathrm{var}}_{l}(0)$ for $x\in[-1/w,1/w]$. Denoting $a=wx/r$, we have $|a|\leq 1/r\leq 1/4$. Taylor’s inequality yields \big{|}\cos(t)-1+\tfrac{t^{2}}{2}-\tfrac{t^{4}}{4!}\big{|}\leq\tfrac{t^{6}}{6!}, for any $|t|\leq 1$. Plugging this bound in the definition of $\Psi^{\mathrm{var}}_{l}(x)$, we get

[TABLE]

Recall that $P_{l}$ has been defined in such a way that $\int_{-r}^{r}P_{l}(\xi)\xi^{2}d\xi=0$. It then follows that

[TABLE]

where we have used that $r>4$, $|a|\leq 1/4$ and the bounds (167). We have proved (174).

∎

D.2.4 Proof of Corollary 6

We first state the following analysis of the test $T^{C,\mathrm{var}}_{\alpha,k_{0}}$.

Corollary 7**.**

Fix any $\xi\in(0,1)$. There exist positive constants $c$, $c^{\prime}$, $c^{\prime\prime}_{\alpha,\beta,\xi}$ and $c^{\prime\prime\prime}_{\alpha,\beta,\xi}$ such that the following holds. Consider any $k_{0}\leq n^{1-\xi}$ and $n\geq c$. Then, for any $\theta\in\mathbb{B}_{0}[k_{0}]$, one has

[TABLE]

Moreover, $\operatorname{\mathbb{P}}_{\theta,\sigma}[T^{C,\mathrm{var}}_{\alpha,k_{0}}=1]\geq 1-\beta-\frac{2\|\theta\|_{1}}{n^{4}\sigma_{+}}-\frac{2}{n^{3}}$ for any vector $\theta$ satisfying $\|\theta\|_{0}\leq c^{\prime}n$ and

[TABLE]

Also, $\operatorname{\mathbb{P}}_{\theta,\sigma}[T^{C,\mathrm{var}}_{\alpha,k_{0}}=1]\geq 1-\beta-\frac{2\|\theta\|_{1}}{n^{4}\sigma_{+}}-\frac{2}{n^{3}}$ for any vector $\theta$ satisfying

[TABLE]

In the sequel, $\operatorname{\mathbb{P}}_{U}$ stands for the probability with respect to $U$. As we did for $Y$, we denote $\mathcal{S}[U,\theta]$ denote the coordinates $i$ such that $|\theta_{i}|>(U+1)\sigma_{+}n^{2}$. Also, we write $\widetilde{Y}(\mathcal{S}[U,\theta]):=(Y_{i}),i\in([n]\setminus\mathcal{S}[U,\theta])$ and $\widetilde{\theta}(\mathcal{S}[U,\theta]):=(\theta_{i}),i\in([n]\setminus\mathcal{S}[U,\theta])$. Note first that $\|\widetilde{\theta}(\mathcal{S}[U,\theta])\|_{1}\leq 2\sigma_{+}n^{3}$. Let us call $\overline{T}^{C,U}_{\alpha,k_{0}-|\mathcal{S}(U,\theta)|}$ the oracle test which is applied to the size $n-|\mathcal{S}(U,\theta)|$ vector $\widetilde{Y}(\mathcal{S}(U,\theta))$ when $k_{0}\geq|\mathcal{S}(U,\theta)|$. Conditionally on $U=u$ :

• •

If $\theta\in\mathbb{B}_{0}[k_{0}]$ then $|\mathcal{S}[u,\theta]|\leq k_{0}$, then $\widetilde{\theta}(\mathcal{S}[u,\theta])\in\mathbb{B}_{0}[k_{0}-|\mathcal{S}[U,\theta|]$. We know from Corollary 7 that

[TABLE]

• •

If $|\mathcal{S}[U,\theta]|>k_{0}$, then the test reject the null with probability one. Consider the case where $|\mathcal{S}[U,\theta]|\leq k_{0}$. If $\theta$ satisfies (65), then $\widetilde{\theta}(\mathcal{S}[u,\theta])$ satisfies the counterpart of Condition (181) for a test of sample size $n-|\mathcal{S}[U,\theta|$. Hence, it follows from Corollary 7 that

[TABLE]

Similarly, the test rejects with high probability when Condition (66) is satisfied.

Integrating these bounds with respect to $\operatorname{\mathbb{P}}_{U}$, we conclude that the type I error probability of the oracle test is smaller than $\alpha+6/n$. Besides, for any $\theta$ satisfying either (65) or (66), the probability of rejection is larger than $1-\beta-6/n$.

It remains to prove that the trimmed test $\overline{T}^{C,\mathrm{var}}_{\alpha,k_{0}}$ agrees with the oracle test $\overline{T}^{C,U}_{\alpha,k_{0}-|\mathcal{S}(U,\theta)|}$ except on an event of small probability.

Lemma 19**.**

Fix any $\theta\in\mathbb{R}^{n}$. Define the events $\mathcal{E}$ and $\mathcal{E}^{\prime}$ by

[TABLE]

Then, $\operatorname{\mathbb{P}}_{\theta,\sigma}[\mathcal{E}]\geq 1-1/n$ and $P_{U,\epsilon}(\mathcal{E}^{\prime})\geq 1-4\sqrt{\log(n)}/n$.

Proof of Lemma 19.

It follows from the Gaussian concentration inequality together with an union bound that $\operatorname{\mathbb{P}}_{\theta,\sigma}[\mathcal{E}]\geq 1-1/n$. Turning to $\mathcal{E}^{\prime}$, we observe the probability of the event

[TABLE]

is less than $4\sqrt{\log(n)}/n^{2}$. Taking an union bound over all $i$, we conclude that $\mathcal{P}[\mathcal{E}^{\prime}]\geq 1-4\log(n)/n$. ∎

Note that $\mathcal{E}\cap\mathcal{E}^{\prime}\subset\{\mathcal{S}[U;Y]=\mathcal{S}[U;\theta]\}$. As a consequence, outside an event of probability less than $5\sqrt{\log(n)}/n$, the oracle test and the trimmed test agree. This concludes the proof.

Proof of Corollary 7.

This corollary is a direct consequence of Theorems 5, 6 and 7. The constants $C$ in Theorems 5 and 7 are chosen large enough so that when Conditions (52) or Conditions (62) are not satisfied, then Condition (57) in Theorem 6 is met. We focus on (65), the result (66) being proved similarly.

Case $k_{0}\leq\sqrt{n}$ and $\Delta\leq\sqrt{n}$. If Condition (52) holds, then the bound follows from Theorem 5. If Condition (52) does not hold, then the test $T_{\alpha/3,k_{0}}^{B}$ rejects the null hypothesis with high probability by Theorem 6.

Case $k_{0}\leq\sqrt{n}$ and $\Delta\geq\sqrt{n}$. Theorem 6 leads to the desired bound.

Case $k_{0}\geq\sqrt{n}$ and $\Delta\leq\sqrt{k_{0}n^{1/2}}\vee\frac{k_{0}^{2}}{n}$. If Condition (52) is not satisfied, then $T_{\alpha/3,k_{0}}^{B}$ rejects the null hypothesis with high probability. Otherwise, Theorem 5 ensures that the Higher-Criticism test rejects the null hypothesis with high probability if $\theta^{2}_{(k_{0}+\Delta)}$ is large in front of $\log(1+k_{0}/\Delta)$. For $\Delta\in(\sqrt{k_{0}n^{1/2}},\frac{k_{0}^{2}}{n})$ we have

[TABLE]

Case $k_{0}\geq\sqrt{n}$ and $k_{0}\geq\Delta\geq\sqrt{k_{0}n^{1/2}}\vee\frac{k_{0}^{2}}{n}$. Theorem 7 leads to the desired bound if Condition (62) is satisfied. Otherwise, Theorem 6 enforces the desired result.

Case $k_{0}\geq\sqrt{n}$ and $\Delta\geq k_{0}$. This is again a consequence of Theorem 6.

∎