A strong converse bound for multiple hypothesis testing, with applications to high-dimensional estimation

TL;DR

This paper introduces a new technique based on binary hypothesis testing to derive tighter, easily computable lower bounds on minimax risk in statistical inference, improving upon traditional Fano's inequality across various applications.

Contribution

The authors adapt a binary hypothesis testing approach to obtain sharper lower bounds in statistical inference, surpassing Fano's inequality in accuracy and applicability.

Findings

Tighter risk lower bounds in density estimation

Improved bounds in active learning of classifiers

Enhanced bounds in compressed sensing

Abstract

In statistical inference problems, we wish to obtain lower bounds on the minimax risk, that is to bound the performance of any possible estimator. A standard technique to obtain risk lower bounds involves the use of Fano's inequality. In an information-theoretic setting, it is known that Fano's inequality typically does not give a sharp converse result (error lower bound) for channel coding problems. Moreover, recent work has shown that an argument based on binary hypothesis testing gives tighter results. We adapt this technique to the statistical setting, and argue that Fano's inequality can always be replaced by this approach to obtain tighter lower bounds that can be easily computed and are asymptotically sharp. We illustrate our technique in three applications: density estimation, active learning of a binary classifier, and compressed sensing, obtaining tighter risk lower bounds in each case.

Equations

$$\underset{\hat{\theta }}{\inf }\underset{\theta \in \mathcal{F}}{\sup }\mathbb{E}\left[w(d(\theta ,\hat{\theta }))\right],$$

$$\mathbb{P}\left(d(\theta ,\hat{\theta })\ge A{\psi }_n\right)$$

$$\underset{\theta \in \mathcal{F}}{\sup }\mathbb{E}\left[w\left(\frac{1}{{\psi }_n}d(\theta ,\hat{\theta })\right)\right]\ge w(A)\left(\underset{\theta \in \mathcal{F}}{\sup }\mathbb{P}\left(d(\theta ,\hat{\theta })\ge A{\psi }_n\right)\right).$$

$$d(\theta_{i},\theta_{j})\geq d_{\min},\mbox{ \;\;\; for all $i\neq j$.}$$

$$\underset{\theta \in \mathcal{F}}{\sup }\mathbb{P}\left(d(\theta ,\hat{\theta })\ge A{\psi }_n\right)$$

$$\ge \underset{i\in \{1,\dots ,M\}}{\max }\mathbb{P}({\theta }_{\hat{i}}\ne {\theta }_i)$$

$$\ge \underset{i\in \{1,\dots ,M\}}{\max }\mathbb{P}({\theta }_{i^{\ast }}\ne {\theta }_i)$$

$$\ge \frac{1}{M}\sum _{i=1}^M\mathbb{P}({\theta }_{i^{\ast }}\ne {\theta }_i)=:{ϵ}_M.$$

$$\underset{\hat{\theta }}{\inf }\underset{\theta \in \mathcal{F}}{\sup }\mathbb{E}\left[w\left(\frac{1}{{\psi }_n}d(\theta ,\hat{\theta })\right)\right]\ge w(A){ϵ}_M.$$

$${ϵ}_M\ge 1-\frac{\log 2+\frac{1}{M}{\sum }_{i=1}^{M}D(P_{{\theta }_i}\Vert \overline{P})}{\log M},$$

$$\frac{1}{M}\sum _{i=1}^{M}D(P_{{\theta }_i}\Vert \overline{P})\le \alpha \log M,$$

$$H_0:(\theta ,\mathbf{Y})$$

$$H_1:(\theta ,\mathbf{Y})$$

$${ϵ}_M\ge 1-\frac{(1+\lambda )}{{\left(\lambda M\right)}^{\frac{\lambda }{1+\lambda }}}{\left[\sum _{i=1}^M\frac{1}{M}\exp \left(\lambda D_{1+\lambda }(P_{\mathbf{Y}|{\theta }_i}\Vert Q_{\mathbf{Y}})\right)\right]}^{\frac{1}{1+\lambda }}.$$

$$D_{1+\lambda }(P_{\mathbf{Y}|{\theta }_i}\Vert Q_{\mathbf{Y}}):=\frac{1}{\lambda }\log \left({\int }_{\mathcal{Y}}{\left(\frac{d{P}_{\mathbf{Y}|{\theta }_i}}{d{Q}_{\mathbf{Y}}}\right)}^{1+\lambda }d{Q}_{\mathbf{Y}}\right).$$

$$\frac{1}{M}\ge \frac{1}{\gamma }\left(1-{ϵ}_M-P_{\theta \mathbf{Y}}\left[\frac{d{P}_{\mathbf{Y}|\theta }}{d{Q}_{\mathbf{Y}}}>\gamma \right]\right).$$

$$P_{\theta \mathbf{Y}}\left[\frac{d{P}_{\mathbf{Y}|\theta }}{d{Q}_{\mathbf{Y}}}>\gamma \right]={\int }_{\mathcal{Y}}\sum _{i=1}^M\frac{1}{M}\mathbb{I}(\theta ={\theta }_i)\frac{d{P}_{\mathbf{Y}|\theta }}{d{Q}_{\mathbf{Y}}}(\mathbf{y})\mathbb{I}\left[\frac{d{P}_{\mathbf{Y}|\theta }}{d{Q}_{\mathbf{Y}}}(\mathbf{y})>\gamma \right]d{Q}_{\mathbf{Y}}(\mathbf{y})$$

$$={\int }_{\mathcal{Y}}\sum _{i=1}^M\frac{1}{M}\frac{d{P}_{\mathbf{Y}|{\theta }_i}}{d{Q}_{\mathbf{Y}}}(\mathbf{y})\mathbb{I}\left[\frac{d{P}_{\mathbf{Y}|{\theta }_i}}{d{Q}_{\mathbf{Y}}}(\mathbf{y})>\gamma \right]d{Q}_{\mathbf{Y}}(\mathbf{y})$$

$$\displaystyle\quad\leq\int_{\mathcal{Y}}\sum_{i=1}^{M}\frac{1}{M}\,\frac{dP_{{\mathbf{Y}}|\theta_{i}}}{dQ_{{\mathbf{Y}}}}({\mathbf{y}})\,\left(\frac{1}{\gamma}\frac{dP_{{\mathbf{Y}}|\theta_{i}}}{dQ_{{\mathbf{Y}}}}({\mathbf{y}})\right)^{\lambda}dQ_{{\mathbf{Y}}}({\mathbf{y}})\mbox{ \;\;\;\;\; for $\lambda>0$.}$$

$$\frac{1}{M}\ge \underset{\gamma >0}{\sup }\left[\frac{1-{ϵ}_M}{\gamma }-\frac{1}{{\gamma }^{1+\lambda }}\sum _{i=1}^M\frac{1}{M}{\int }_{\mathcal{Y}}{\left(\frac{d{P}_{\mathbf{Y}|{\theta }_i}}{d{Q}_{\mathbf{Y}}}(\mathbf{y})\right)}^{1+\lambda }d{Q}_{\mathbf{Y}}(\mathbf{y})\right].$$

$$q^{\ast }(\mathbf{y})=\frac{1}{C}{\left(\sum _{i=1}^M\frac{1}{M}(p_{{\theta }_i}(\mathbf{y}){)}^{1+\lambda }\right)}^{\frac{1}{1+\lambda }},$$

$${\int }_0^1\theta (x)dx=1,a_0\le \theta (x)\le a_1<\infty ,\left|{\theta }^{\prime \prime }(x)\right|\le a_2,x\in \mathbb{R},$$

$$d(\theta ,{\hat{\theta }}_n)={\int }_0^1{\left(\sqrt{\theta (x)}-\sqrt{{\hat{\theta }}_n(x)}\right)}^{2}dx.$$

$${\int }_0^{1}g(x)dx=0\text{mbox}and{\int }_0^1{\left(g(x)\right)}^{2}dx=a.$$

$$g_{j}(x)=\frac{c}{m^{2}}g(mx-j){\mathbb{I}}\left(\frac{j}{m}\leq x<\frac{j+1}{m}\right),\mbox{\;\;\; for $j=0,\ldots,m-1$ }.$$

$${\mathcal{M}}_m=\left\{f_{\mathbf{\tau }}(y)=1+\sum _{j=0}^{m-1}{\tau }_j{g}_j(y)\right\}.$$

$${\mathcal{P}}_{M,\frac{a{c}^2}{3m^4}}=\{f_{\mathbf{\tau }}:\mathbf{\tau }\in \mathcal{A}\}\subseteq {\mathcal{M}}_m,$$

$${\left[\sum _{i=1}^M\frac{1}{M}{\int }_{\mathcal{Y}}{\left(\frac{d{P}_{\mathbf{Y}|{\theta }_i}}{d{Q}_{\mathbf{Y}}}(\mathbf{y})\right)}^{2}d{Q}_{\mathbf{Y}}(\mathbf{y})\right]}^{\frac{1}{2}}$$

$$\le \exp \left(\frac{c^{2}an}{2m^4}\right).$$

$$\underset{\theta \in \mathcal{F}}{\sup }\mathbb{E}d({\hat{\theta }}_n,\theta )\ge \frac{c^{2}a{\nu }^4}{6}{n}^{-4/5}{ϵ}_M,$$

$${ϵ}_M\ge 1-2\exp \left(-\frac{n^{1/5}}{2\nu }\left(c_0-{\nu }^5{c}^{2}a\right)\right).$$

$$\underset{\theta \in \mathcal{F}}{\sup }\mathbb{E}d({\hat{\theta }}_n,\theta )\ge \frac{c_0^{4/5}(c^{2}a{)}^{1/5}}{6}{n}^{-4/5}(1-o(1)).$$

$$\underset{j}{\max }\mathbb{E}d(\hat{\theta },{\theta }_j)\ge {\psi }_n{ϵ}_M=\frac{c^{2}a}{6m^4}{ϵ}_M.$$

$${ϵ}_M$$

Full text

1 Introduction

When solving an inference problem, we would like to know if the algorithm we use is close to optimal. In statistical language we seek to give a lower bound on the performance of any estimator over a class of problems (often called the minimax risk over the class). In the language of information theory, we speak of converse results, which give performance bounds for all communication schemes over a noisy channel.

In the statistics literature, one standard approach to proving converse results is via Fano’s inequality (see [1, Theorem 2.11.1]). However, recent information-theoretic literature has shown how to obtain sharper converse bounds. The resulting improvements can be significant at finite sample size, and give bounds that are close to optimal, as illustrated in the work of Polyanskiy, Poor and Verdú [2]. The present paper shows how the method of [2], although developed for channel coding problems, gives stronger risk lower bounds for high-dimensional estimation problems, compared to the standard Fano approach.

We first describe the general set-up, following the treatment and notation of [3, Chapter 2]. Consider an inference problem (possibly non-parametric) where we wish to estimate some $\theta\in{\mathcal{F}}$ from samples ${\mathbf{Y}}=(Y_{1},\ldots,Y_{n})$ generated according to a distribution $P_{\theta}({\mathbf{Y}})$. For example, in Section 3 we consider $\theta$ to be a probability density chosen from a pre-specified class, and in Section 5 we consider $\theta$ to be a $k$-sparse vector in ${\mathbb{R}}^{n}$. Let $\widehat{\theta}:=\widehat{\theta}({\mathbf{Y}})$ be any estimator of $\theta$ and let $d(\theta,\widehat{\theta})$ represent the loss. We assume that $d$ is a distance, although (as in [3]) our results hold when $d$ is a semi-distance; that is, when $d(\theta,\theta^{\prime})=0$ need not imply that $\theta=\theta^{\prime}$. We obtain lower bounds on the minimax risk

[TABLE]

where $w$ is any monotonically increasing function with $w(0)=0$. For example, we may consider $w(u)=u^{p}$ for any $p>0$ or $w(u)={\mathbb{I}}(u\geq c)$ for some $c>0$.

A standard approach for obtaining a lower bound on (1) is as follows. First, a set $\{\theta_{1},\ldots,\theta_{M}\}$ $\subseteq{\mathcal{F}}$ is chosen, with a lower bound on the pairwise distance between any two of its elements, where the distance is measured using the loss function $d(\cdot,\cdot)$. Then, any estimator $\widehat{\theta}$ defines an $M$-ary hypothesis test that detects one of $\{\theta_{1},\ldots,\theta_{M}\}$ based on the data $\mathbf{Y}$. Next, the key step is to obtain a lower bound for the error probability associated with this hypothesis test. For a well-constructed set, Fano’s inequality often shows that this average error probability is bounded away from [math] as $n\to\infty$. In this paper, we present a technique that often shows that it approaches $1$ as $n\to\infty$. In information theory parlance (see for example [1, P.207]), we prove a “strong converse” result in contrast to the “weak converse” provided by Fano’s inequality.

We now explain the details, following the framework in [3] (see also [4], [5]). For any positive constants $A$ and $\psi_{n}$, using Markov’s inequality we have

[TABLE]

which implies

[TABLE]

When applying (2), we typically choose $\psi_{n}$ as a decreasing function of $n$ to give the desired convergence rate, and $A$ as a constant that can be used to optimize the lower bound. The goal then is to control the bracketed term on the RHS of (2) to obtain a lower bound on the minimax risk. We use the following definition.

Definition 1.1**.**

A collection ${\mathcal{P}}_{M,d_{\min}}=\{\theta_{1},\ldots,\theta_{M}\}\subseteq{\mathcal{F}}$ is called a packing set of size $M$ and minimum distance $d_{\min}$ if

[TABLE]

In general, the packing set is not explicitly constructed, but its existence is guaranteed via combinatorial arguments. In Remarks 3.1 and 4.1 below, existence of packing sets is guaranteed by applying the Gilbert–Varshamov bound. In Remark 5.1, the existence of a packing set is guaranteed via the probabilistic method. We emphasize that we use these existing packing set constructions: our contribution is to provide tighter lower bounds than can be obtained using Fano’s inequality. It is possible that the resulting risk lower bounds could be improved by a further constant factor, by varying the packing set construction.

In statistical language, we think of the packing set ${\mathcal{P}}_{M,d_{\min}}$ as multiple hypotheses to be distinguished on the basis of data. An alternative information-theoretic interpretation is to think of ${\mathcal{P}}_{M,d_{\min}}$ as a codebook, that is a collection of $M$ codewords, one of which is transmitted over a noisy communication channel. Given a packing set ${\mathcal{P}}_{M,d_{\min}}$, any estimator $\widehat{\theta}$ provides a way to distinguish between multiple hypotheses (act as a channel decoder) as follows: given $\widehat{\theta}$, we choose $\hat{i}=\operatorname*{arg\,min}_{j}d(\widehat{\theta},\theta_{j})$, i.e. the index of the closest value in the packing set. In coding theory, this is called the minimum distance decoder.

If $\theta_{i}$ is the true value, a simple triangle inequality argument shows that $\left\{\hat{i}\neq i\right\}\Rightarrow\left\{d(\theta_{i},\widehat{\theta})\geq d_{\min}/2\right\}$. Taking $d_{\min}=2A\psi_{n}$, we can bound the bracketed term on the RHS of (2) by the average error probability $\epsilon_{M}$ of the optimal decoder $i^{*}=i^{*}({\mathbf{Y}})$, since

[TABLE]

This calculation and argument are standard in the literature (see for example [3, Eq. (2.9)], [5, Corollary 2.19]). By substituting (3) in (2), we deduce

[TABLE]

Our main focus is to obtain a sharp and easily computable bound for $\epsilon_{M}$. A standard technique, dating back to Ibragimov and Khasminskii [6], is to bound $\epsilon_{M}$ using Fano’s inequality, which gives the bound [3, Lemma 2.10]

[TABLE]

where ${\overline{P}}:=\frac{1}{M}\sum_{i=1}^{M}P_{\theta_{i}}$, and $D(P\|Q)$ is the Kullback–Leibler (KL) divergence. To apply (5), one typically obtains a bound of the form

[TABLE]

for some constant $\alpha\in(0,1)$ (see [3, Section 2.7.1]). Then (5) implies that $\epsilon_{M}\geq 1-\alpha-\frac{\log 2}{\log M}$, which converges to $(1-\alpha)$ for large sample sizes $n$ (assuming $\log M\to\infty$ as $n\to\infty$), meaning that we deduce a weak converse, and (3) gives a lower bound on the risk (via (2)).

The remainder of the paper is organized as follows. In Section 2, we derive a lower bound on $\epsilon_{M}$ (Theorem 1) that strengthens Fano’s inequality. In Section 2.1, we discuss related prior work. We then apply Theorem 1 to three high-dimensional estimation problems, in each case showing the average error probability $\epsilon_{M}\to 1$ as $n\to\infty$ (strong converse). In each case, our method replaces the Fano-based part of the argument which gives a weak converse. In Section 3, we give a strong converse for a density estimation problem studied by Yu [7]. In Section 4, we obtain strengthened risk lower bounds for active learning of a binary classifier, following Castro and Nowak [8]. In Section 5, we use Theorem 1 to improve (by a factor of nearly 8) lower bounds of Candès and Davenport [9] for the minimax mean-squared error in compressed sensing.

2 Lower bound on the Average Error Probability

We bound the average error probability $\epsilon_{M}$ in (3) using a different binary hypothesis testing problem. Adopting the formalism of [2], we consider a random variable $S$ representing a message chosen uniformly at random from $\{1,\ldots,M\}$. The message $S$ is acted on by the simple encoder that generates codeword $\theta=\theta_{S}$, giving an induced distribution $\pi_{\theta}$ uniform over the set $\{\theta_{1},\ldots,\theta_{M}\}$.

We think of ${\mathbf{Y}}=(Y_{1},\ldots,Y_{n})$ as the output of a channel with input $\theta$. Using arguments from [2, 10], we bound the desired average error probability of the optimal decoder (3) in terms of the Type I error probability of the following binary hypothesis testing problem:

[TABLE]

for some probability distribution $Q_{{\mathbf{Y}}}$ that does not depend on $\theta$. In other words, we wish to determine whether $\theta$ and ${\mathbf{Y}}$ are independent, or are generated by the true underlying channel model. We assume that the measure $Q_{{\mathbf{Y}}}$ dominates $P_{{\mathbf{Y}}|\theta_{i}}$ for $1\leq i\leq M$, and hence the Radon-Nikodym derivative $\frac{dP_{{\mathbf{Y}}|\theta_{i}}}{dQ_{{\mathbf{Y}}}}$ exists.

The space of ${\mathbf{Y}}$ is denoted by $\mathcal{Y}$. Throughout the paper, we use boldface notation to denote vectors of length $n$.

Theorem 1**.**

Let $\epsilon_{M}$ denote the average error probability of any decoder over channel $P_{{\mathbf{Y}}|\theta}$, for a channel code with input distribution $\pi_{\theta}$ uniform over the $M$ codewords $\{\theta_{1},\ldots,\theta_{M}\}$. For any $\lambda>0$, and any distribution $Q_{{\mathbf{Y}}}$ over $\mathcal{Y}$ such that $P_{{\mathbf{Y}}|\theta_{i}}$ is absolutely continuous with respect to $Q_{{\mathbf{Y}}}$ for $1\leq i\leq M$,

[TABLE]

Here $D_{1+\lambda}(P_{{\mathbf{Y}}|\theta_{i}}\|Q_{{\mathbf{Y}}})$ is the Rényi divergence of order $(1+\lambda)$ defined as

[TABLE]

The proof uses the following lemma, itself proved in Appendix A.

Lemma 2.1**.**

With the assumptions and notation of Theorem 1 we have for any $\gamma>0$

[TABLE]

Proof of Theorem 1.

Writing ${\mathbb{I}}(\cdot)$ for the indicator function, the probability in (10) satisfies

[TABLE]

Using this bound (11) in Lemma 10, we have

[TABLE]

Computing the maximum over $\gamma>0$ and rearranging, we get (8). ∎

Remark 2.1**.**

As shown in the active learning example in Sec. 4, one can use upper bounds for the Rényi divergence in (9) to obtain lower bounds for $\epsilon_{M}$. Such upper bounds can found, for example, in [11, 12].

Remark 2.2**.**

In Appendix B, we show how Fano’s inequality in (5) can be obtained from the lower bound on Theorem 1. Furthermore, the examples in the next three sections show that Theorem 1 yields strictly better lower bounds than the Fano-based approach.

Remark 2.3**.**

If we assume that each $P_{{\mathbf{Y}}|\theta_{i}}$ has a density $p_{\theta_{i}}({\mathbf{y}})$ with respect to a common reference measure $\mu$, then the choice of $Q_{{\mathbf{Y}}}$ that maximizes the lower bound in (8) has the following density with respect to $\mu$[13, 14]:

[TABLE]

where the normalizing constant $C=\int_{\mathcal{Y}}\,\left(\sum_{i=1}^{M}\frac{1}{M}(p_{\theta_{i}}({\mathbf{y}}))^{1+\lambda}\right)^{\frac{1}{1+\lambda}}d\mu({\mathbf{y}})$. However, the bound in (8) is generally not computable with this choice of $Q_{{\mathbf{Y}}}$. As we will see in the following sections, the structure of the problem often suggests a natural choice for $Q_{{\mathbf{Y}}}$ that yields a computable lower bound.

2.1 Related work

In [3, Proposition 2.2], Tsybakov gives a result similar to Lemma 10. This result can then be used to obtain a lower bound on $\epsilon_{M}$ using the average pairwise $\chi^{2}$-distance between $q$ and the elements of the packing set [3, Theorem 2.6]. This bound is similar to the one obtained by using $\lambda=1$ in Theorem 1. In this paper, we show that via two examples (active learning and compressed sensing) that Theorem 1 can be applied with a general $\lambda>0$ to obtain stronger non-asymptotic bounds. Furthermore, as $n\to\infty$, Theorem 1 gives a strong converse ($\epsilon_{M}\to 1$), unlike Fano’s inequality.

Birgé [15] gives stronger, but less transparent, bounds than Fano’s inequality using Fano-type arguments; again, $\epsilon_{M}$ is bounded in terms of an average of Kullback–Leibler divergences, but these are used as the argument for a function, rather than directly substituted. Sason and Verdú [16, Section 3] recently derived a generalized Fano’s inequality in terms of the Arimoto-Rényi conditional entropy. They also obtained upper bounds on $\epsilon_{M}$ in terms of the pairwise Rényi divergences [16, Section 4].

Note that an alternative approach to hypothesis testing bounds, avoiding the use of Fano’s inequality, is given by Assouad [17]. Indeed, [7] makes a detailed comparison between Fano-based bounds and those coming from Assouad’s Lemma [17], finding little practical difference. Indeed [7] quotes Birgé [18, p. 279]: “[Fano] is in a sense more general because it applies in more general situations. It could also replace Assouad’s Lemma in almost any practical case …”.

Using Fano’s inequality, Yang and Barron [19] obtained order-optimal minimax risk lower bounds that depend only on global metric entropy features of the underlying function class, without explicitly constructing a packing set. The required metric entropy features (bounds on the packing number and covering number) are available from results in approximation theory for many function classes of interest. Guntuboyina [20] obtained a lower bound on the average error probability in terms of general $f$-divergences, and also generalized the metric entropy results of Yang and Barron [19] to certain $f$-divergences such as the $\chi^{2}$-divergence. An interesting direction for future work would be to obtain a non-asymptotic result analogous to Theorem 1 for the case where only the global metric entropy features are available.

An important historical remark is that Hayashi and Nagaoka [21] first linked channel coding and binary hypothesis testing, with later work [22] by Hayashi clarifying this approach and Nagaoka [23] using similar ideas to derive strong converse results. The recent work by Vazquez-Vilar et al. [10] also provides results characterizing the average error probability of channel coding in terms of the Type I error of a binary hypothesis test. This link with channel coding has been used in other contexts to prove strong converse results, including [24], which derived strong converse results for the group testing problem.

3 Application to density estimation

For the remainder of this paper, we show how Theorem 1 can be applied to a number of high-dimensional estimation problems. In this section we apply Theorem 1 to the following density estimation problem taken from Yu [7, Example 2, P.431]. Let $\mathcal{F}$ be the class of smooth densities on $[0,1]$ such that for any density $\theta\in\mathcal{F}$, we have

[TABLE]

for some positive constants $a_{0},a_{1},a_{2}$. The goal is to estimate the density $\theta$ from ${\mathbf{Y}}=(Y_{1},\ldots,Y_{n})$, where $\{Y_{i}\}$ are generated IID from $\theta$. We want to bound from below the risk of any estimator $\widehat{\theta}_{n}=\widehat{\theta}_{n}({\mathbf{Y}})$, where the loss is measured using squared Hellinger distance, i.e.,

[TABLE]

The packing set in [7] is constructed via a hypercube class of densities defined via small perturbations of the uniform density on $[0,1]$. Fix a smooth, bounded function $g(x)$ with

[TABLE]

We partition the unit interval $[0,1]$ into $m$ subintervals of length $1/m$, and perturb the uniform density on each subinterval by a small amount, proportional to a version of $g$ rescaled and translated to lie on that subinterval. That is, for some sufficiently small fixed constant $c$, we can define the functions

[TABLE]

Considering perturbations of the uniform density by $\pm\{g_{j}\}$, define the following hypercube class of joint densities indexed by ${\boldsymbol{\tau}}=\left(\tau_{1},\ldots,\tau_{m}\right)\in\{\pm 1\}^{m}$:

[TABLE]

The bandwidth parameter $m$ will be chosen later as an increasing function of $n$, to optimize the risk lower bound.

Remark 3.1** (Packing set construction).**

[7, Lemma 4]** There exists a subset ${\mathcal{A}}\subseteq\{-1,1\}^{m}$ of size $M\geq\exp(c_{0}m)$, where $c_{0}\simeq 0.082$, whose elements have minimum pairwise Hamming distance at least $m/3$. It is then shown in [7] that this results in a packing set of densities

[TABLE]

where $ac^{2}/(3m^{4})$ is a lower bound on the squared Hellinger distance (see (13)) between distinct densities in the packing set. Here $a$ is defined in (14), and $c$ is defined in (15). We use exactly this packing set ${\mathcal{P}}_{M,ac^{2}/(3m^{4})}$ as the set of $M$ codewords $\{\theta_{1},\ldots,\theta_{M}\}$ in Theorem 1.

We now use Theorem 1 to bound the risk. To do this, we first state an explicit bound (to be proved in Appendix C) on the bracketed term in (8), for $\lambda=1$.

Lemma 3.1**.**

Taking $Q_{{\mathbf{Y}}}$ to be the uniform measure on $[0,1]^{n}$ and identifying each $\frac{dP_{{\mathbf{Y}}|\theta_{i}}}{dQ_{{\mathbf{Y}}}}$ with a density $f_{{\boldsymbol{\tau}}}^{n}({\mathbf{y}})=\prod_{i=1}^{n}f_{{\boldsymbol{\tau}}}(y_{i})$ for $\tau\in{\mathcal{A}}$, with $\lambda=1$, the bracketed term in (8) becomes:

[TABLE]

Combining Lemma 3.1 with Theorem 1, we deduce the following lower bound.

Proposition 3.2**.**

For any positive constant $\nu<(c_{0}/(c^{2}a))^{1/5}$, the risk of any estimator $\widehat{\theta}_{n}$ satisfies

[TABLE]

where

[TABLE]

Therefore, for large $n$ we have

[TABLE]

Proof.

We apply Theorem 1 by setting the minimum distance $ac^{2}/(3m^{4})$ of the packing set in Remark 3.1 to $2A\psi_{n}$. Taking $A=1$, we obtain $\psi_{n}=c^{2}a/(6m^{4})$. Taking $w$ to be the identity in (4), we deduce

[TABLE]

Taking $\lambda=1$ in Theorem 1 and using Lemma 3.1, we bound $\epsilon_{M}$ as

[TABLE]

where $(a)$ is obtained using the fact that the packing set of densities ${\mathcal{P}}_{M,ac^{2}/(3m^{4})}$ has size $M\geq\exp(c_{0}m)$, as described in Remark 3.1 above. We therefore have

[TABLE]

The result (17) follows by taking $m=n^{1/5}/\nu$.

To obtain the asymptotic bound in (19), we take $\nu$ to approach $(c_{0}/(c^{2}a))^{1/5}$ as $n\rightarrow\infty$, but slowly enough that ensure that the exponent on the RHS of (18) is negative so that $\epsilon_{M}$ tends to 1. For example, we can take $\nu=\left(\frac{c_{0}}{c^{2}a}\right)^{1/5}(1-n^{-1/\kappa})$ for $\kappa>25$. ∎

Remark 3.2**.**

The paper [7] derives Fano-type bounds in this setting: combining Lemmas 3 and 5 of [7] and taking taking $m=n^{1/5}/\nu$ gives the same bound as (17), but with a looser lower bound on $\epsilon_{M}$ given by

[TABLE]

For the bound (21) to be meaningful, we need $\nu<\left(\frac{c_{0}}{c^{2}a}\frac{1-c_{g}}{2}\right)^{1/5}$, where $c_{g}=c\sup_{x}\left\lvert g(x)\right\rvert$. The scaling factor $c$ has to be chosen so that $c_{g}<1$.

Thus Proposition 3.2 provides a strong converse (error probability tending to $1$), whereas the result (21) extracted from [7] gives a weak converse (error probability bounded away from [math]). Our bound also offers greater flexibility in choosing $\nu$ and removes the need to control $c_{g}$.

Remark 3.3**.**

Theorem 1 can similarly be applied to obtain strong converses for estimating densities belonging to either Hölder or Sobolev classes, strengthening the risk lower bounds described in [3, Sec. 2.6.1]

4 Application to active learning of a classifier

In this section, we use Theorem 1 to derive strengthened minimax lower bounds for active learning algorithms for a family of classification problems introduced by Castro and Nowak [8] (see also [25]). We use the explicit packing set construction of [8], but modify their notation for consistency.

Consider data of the form ${\mathbf{Y}}=({\mathbf{U}},{\mathbf{V}})=\left((U_{1},V_{1}),\ldots,(U_{n},V_{n})\right)$. Each pair $(U_{r},V_{r})$ consists of a feature vector $U_{r}\in{\mathbb{R}}^{d}$ (where we assume $d\geq 2$) and a binary label $V_{r}\in\{0,1\}$, and is drawn independently from an underlying joint distribution $P_{UV}=P_{U}P_{V|U}$. The goal of classification is to predict the value of label $V$, given a future $U$ observation. This is done via $G$, a measurable subset of ${\mathbb{R}}^{d}$. Given a $U\in\mathbb{R}^{d}$, the classifier estimates its label as $\hat{V}:=\hat{V}(U):={\mathbb{I}}(U\in G)$. The risk of a classifier is the probability of classification error, given by

[TABLE]

where $(U,V)\sim P_{UV}$. It is well-known (see [25]) that, given knowledge of $P_{UV}$, the Bayes-optimal classifier is

[TABLE]

where the feature conditional probability $\eta(u)=P_{V|U}(1|u)$. As $P_{UV}$ is unknown, our goal is to estimate $G^{*}$ from data ${\mathbf{Y}}$. The performance of classifier $\widehat{G}_{n}$ is measured by excess risk (or regret) [8, Eq. (1)]

[TABLE]

where $\Delta$ represents the symmetric difference between sets. For the remainder of this section, as in [8], we will assume that $P_{U}$ is supported on $[0,1]^{d}$. It is clear that the difficulty of a classification problem will depend on both the shape of the boundary of $G^{*}$ and the behaviour of $(2\eta(u)-1)$ for $u$ close to this boundary. We consider the class of joint distribution functions $\mathsf{BF}(\alpha,\kappa,L,c)$, defined formally in [8, Section IV]. For our purposes it suffices to understand this class as a set of distributions $P_{UV}$ such that:

The boundary of $G^{*}$ can be expressed as an $\alpha$-Hölder smooth function with constant $L$. 2. 2.

The value of $|\eta(u)-1/2|$ is at least $cD^{\kappa-1}$ for points $u$ at distance $D$ from the boundary, where $\kappa\geq 1$.

Algorithms that attempt to learn the Bayes-optimal classifier $G^{*}$ from data are categorized as passive or active. Passive learning algorithms aim to learn $G^{*}$ from a pre-specified, possibly random, choice of $(U_{1},\ldots,U_{n})$ and the corresponding labels $(V_{1},\ldots,V_{n})$. In contrast, active learning algorithms choose each $U_{r}$ based on previous values $(U,V)_{r}^{-}:=\left(U_{1},\ldots,U_{r-1},V_{1},\ldots,V_{r-1}\right)$. This allows us to adaptively probe the boundary of $G^{*}$. A (randomized) active learning algorithm is defined by a sequence of conditional distributions $P_{U_{r}|(U,V)_{r}^{-}}:=P_{U_{r}|\left(U_{1},\ldots,U_{r-1},V_{1},\ldots,V_{r-1}\right)}$, which defines the joint distribution as follows:

[TABLE]

where $P_{V_{r}|U_{r}}\equiv P_{V|U}$; in particular, conditioned on $U_{r}$, label $V_{r}$ is independent of $(U_{1},\ldots,U_{r-1})$. We assume that for each $r$, the conditional distribution $P_{U_{r}|(U,V)_{r}^{-}}$ has a density $p_{U_{r}|(U,V)_{r}^{-}}$ with respect to Lebesgue measure on $[0,1]^{d}$. Note that active learning algorithms correspond to channel coding with feedback, and to adaptive group testing algorithms [24]. Passive learning corresponds to channel coding without feedback, and to non-adaptive group testing algorithms.

We provide lower bounds on the excess risk of active learning algorithms that strengthen those in [8, Theorem 3], but our techniques can also be applied to [8, Theorem 4], which applies in the passive case. We use the packing set constructed in [8], which is defined via a hypercube class of joint distributions on $(U,V)$. Fix an integer $m$ (to be chosen later as a function of $n$). For each vector ${\boldsymbol{\tau}}\in\{0,1\}^{m^{d-1}}$, Castro and Nowak [8, Appendix C] construct a unique distribution of $(U,V)$ whose feature conditional probability is denoted by $\eta_{{\boldsymbol{\tau}}}(u)$, and the corresponding Bayes classifier is denoted by $G^{*}_{{\boldsymbol{\tau}}}$. We denote this hypercube class of $2^{m^{d-1}}$ distributions by ${\mathcal{F}}_{m}$. Each distribution in ${\mathcal{F}}_{m}$ has the same $U$-marginal $P_{U}$. Thus the joint distribution is determined by the conditional distribution $P_{V|U}$. The conditional distributions in ${\mathcal{F}}_{m}$ (equivalently, the feature conditional probabilities $\eta_{{\boldsymbol{\tau}}}(u)$ for each ${\boldsymbol{\tau}}\in\{0,1\}^{m^{d-1}}$) are not explicitly defined here, but the definition can be found in the displayed equation at the foot of [8, p.2350]. The definition ensures that the hypercube class ${\mathcal{F}}_{m}\subseteq\mathsf{BF}(\alpha,\kappa,L,c)$.

The packing set defined in [8, Appendix C] is a subset of distributions in ${\mathcal{F}}_{m}$.

Remark 4.1** (Packing set construction).**

[8, Lemma 2]** There exists a subset ${\mathcal{A}}\subseteq\{0,1\}^{m^{d-1}}$ of size $M+1$ with $M\geq 2^{m^{d-1}/8}$, whose elements have minimum pairwise Hamming distance at least $m^{d-1}/8$. It is then shown in [8] that this results in a packing set of functions ${\mathcal{P}}_{M+1,\beta_{m}/8}=\{\eta_{{\boldsymbol{\tau}}}:{\boldsymbol{\tau}}\in{\mathcal{A}}\}$, where $\beta_{m}=LHm^{-\alpha}$, and $\eta_{{\boldsymbol{\tau}}}(1|u)=P_{V|U}(u)$. (Hence $1-\eta_{{\boldsymbol{\tau}}}(u)=P_{V|U}(0|u)$.) Here $\beta_{m}/8$ is a lower bound on the set distance between distinct elements of ${\mathcal{P}}_{M+1,\beta_{m}/8}$, defined as

[TABLE]

and $H=\left\lVert h\right\rVert_{1}$ is the norm of a suitable smooth function $h$.

Furthermore, ${\mathcal{P}}_{M+1,\beta_{m}/8}$ contains the function $\eta_{{\boldsymbol{0}}}$, corresponding to the point ${{\boldsymbol{\tau}}}=(0,0,\ldots,0)$ in the hypercube. We use the other $M$ functions in the packing set ${\mathcal{P}}_{M+1,\beta_{m}/8}$ to act as the $M$ codewords $\{\theta_{1},\ldots,\theta_{M}\}$ in Theorem 1. The Bayes classifiers corresponding to these codewords are denoted by $G_{1}^{*},\ldots,G_{M}^{*}$.

As in Section 3, we prove an explicit bound on the bracketed term (8) in Theorem 1, with $({\mathbf{u}},{\mathbf{v}})$ corresponding to ${\mathbf{y}}$ in (8).

Lemma 4.1**.**

For an active learning algorithm described by $\prod_{r=1}^{n}P(U_{r}|(U,V)_{r}^{-})$, we take $Q_{{\mathbf{U}},{\boldsymbol{V}}}({\boldsymbol{U}},{\boldsymbol{V}}):=\prod_{r=1}^{n}P(U_{r}|(U,V)_{r}^{-})\prod_{r=1}^{n}P_{{\mathbf{0}}}(V_{r}|U_{r})$, where $P_{{\mathbf{0}}}(V_{r}|U_{r})$ is the conditional probability mass function determined by $\eta_{{\mathbf{0}}}$ which corresponds to the point ${{\boldsymbol{\tau}}}=(0,0,\ldots,0)$ in the hypercube.

Further, for each ${\boldsymbol{\tau}}\in{\mathcal{A}}$ and ${\boldsymbol{\tau}}\neq{\boldsymbol{0}}$, we can take

[TABLE]

where $P_{{\boldsymbol{\tau}}}(V_{r}|U_{r})$ is the conditional probability mass function determined by $\eta_{{\boldsymbol{\tau}}}$. Then, for any $\lambda\in(0,1]$, the bracketed term in (8) satisfies

[TABLE]

where for brevity we write an integral to represent integration and summation over the product space $\mathcal{Y}_{n}=[0,1]^{d\times n}\otimes\{0,1\}^{n}$, and $\beta_{m}=LHm^{-\alpha}$ as defined in Remark 4.1.

Proof.

See Appendix D. ∎

Combining Lemma 4.1 with Theorem 1, we deduce the following lower bound.

Proposition 4.2**.**

Let $\rho=(d-1)/\alpha$. For any positive constant $\nu$, the risk of a classifier $\widehat{G}_{n}$ learnt via any active learning algorithm satisfies

[TABLE]

where

[TABLE]

for any $\lambda\in(0,1]$. Therefore, for large $n$ we have

[TABLE]

Proof.

Consider the $M$ codewords chosen from the packing set ${\mathcal{P}}_{M+1,\beta_{m}/8}$, as described in Remark 4.1, with corresponding Bayes classifiers $G^{*}_{1},\ldots,G^{*}_{M}$ . The minimum pairwise set distance between these Bayes classifiers is at least $\beta_{m}/8$. Equating the minimum distance of the packing set given by $2A\psi_{n}$ (in Theorem 1) to $\beta_{m}/8$, taking $A=1$ we obtain $\psi_{n}=\beta_{m}/16=LHm^{-\alpha}/16$.

Using Lemma 4.1 in Theorem 1, for any $\lambda\in(0,1]$ the average error probability $\epsilon_{M}$ can be bounded from below as

[TABLE]

where inequality $(a)$ is obtained using the fact that packing set of distributions ${\mathcal{P}}_{M+1,\beta_{m}/8}$ has $M\geq 2^{m^{d-1}/8}$, as described in Remark 4.1 above.

Now, consider any distribution $P_{UV}\in\mathsf{BF}(\alpha,\kappa,L,c)$ with Bayes classifier $G^{*}$. It is shown in [8, p.2351] that the event

[TABLE]

Defining $f(\psi_{n}):=\min\left(\frac{4c}{\kappa 2^{\kappa}}\psi^{\kappa}_{n},\psi_{n}\right)$, we therefore obtain the following chain of inequalities:

[TABLE]

where inequality $(b)$ follows from (3). Hence, using $\psi_{n}=LHm^{-\alpha}/16$, we have111As $m\gg 1$ and $\kappa\geq 1$, we assume for brevity that $f(\psi_{n})=\frac{4c}{\kappa 2^{\kappa}}\psi^{\kappa}_{n}$. This is always true for sufficiently large $m$ when $\kappa>1$. However, if $\kappa=1$ and $c>\frac{1}{2}$, then $f(\psi_{n})=\psi_{n}$; however, the definition of $c$ in [8, Eq. (9)] implies that $c$ can be restricted to $(0,\frac{1}{2}]$ without loss of generality.

[TABLE]

where the last inequality is obtained using (28). The result follows by taking $m=n^{\frac{1}{\alpha(2\kappa-2)+d-1}}/\nu$.

To obtain (27), we choose the supremum of $\nu$ such that $\epsilon_{M}\to 1$ as $n\to\infty$ in order to obtain the largest possible prefactor in (4). ∎

Remark 4.2**.**

The paper [8] derives Fano-type bounds in this setting: in particular, taking $m=n^{\frac{1}{\alpha(2\kappa-2)+d-1}}/\nu$, the computation in p.2351 of [8] together with Theorem 6 of that paper gives the same bound as (25), but with a looser lower bound on $\epsilon_{M}$ given by

[TABLE]

where $\xi=\frac{256}{\log 2}c^{2}(LH)^{2\kappa-2}\nu$. For the bound (30) to be meaningful, we need $\xi<\frac{1}{2}$, which implies $\nu<\frac{\log 2}{512c^{2}(LH)^{2\kappa-2}}$. Again, Proposition 4.2 provides a strong converse, while (30) provides a weak one ($\epsilon_{M}$ bounded away from zero).

5 Application to compressed sensing

We now describe how Theorem 1 can give improved risk lower bounds in compressed sensing. The goal in compressed sensing is to estimate a sparse vector $\mathbf{x}\in\mathbb{R}^{n}$ from a measurement $\mathbf{y}\in\mathbb{R}^{m}$ of the form

[TABLE]

Here $\mathbf{A}\in\mathbb{R}^{m\times n}$ is the (known) measurement matrix, and ${\mathbf{w}}\sim\mathcal{N}(0,\sigma^{2}\mathbf{I}_{m})$ is the noise vector. Throughout this section $\left\lVert{\mathbf{x}}\right\rVert$ denotes the $L_{2}$ Euclidean norm of a vector ${\mathbf{x}}$, and $\left\lVert\mathbf{A}\right\rVert_{F}$ denotes the Frobenius norm of a matrix $\mathbf{A}$, defined by $\left\lVert\mathbf{A}\right\rVert_{F}^{2}=\textsf{Tr}\left(\mathbf{A^{T}}\mathbf{A}\right)$. We assume that the signal ${\mathbf{x}}$ is $k$-sparse, by considering ${\mathbf{x}}\in\Sigma_{k}$, where

[TABLE]

In the pioneering works [26, 27, 28] of Candès, Donoho, Romberg, and Tao, among others, it was shown that under suitable assumptions on ${\mathbf{A}}$ and the sparsity level $k$, the signal could be efficiently estimated to a high degree of accuracy, even when $m\ll n$. For example, when $\mathbf{A}$ satisfies the Restricted Isometry Property [29], reconstruction techniques based on minimizing the $L_{1}$ norm produce an estimate $\hat{{\mathbf{x}}}$ which satisfies

[TABLE]

with high probability, provided that $m$ is of order at least $k\log(n/k)$ [30]. ($C_{0}$ is a universal positive constant.)

To complement these achievability results, several authors, e.g., [31, 32, 33, 9] have derived lower bounds on the minimax risk under various assumptions on ${\mathbf{A}}$ and ${\mathbf{x}}$. The minimax risk is defined as

[TABLE]

We show how Theorem 1 can be used to obtain a strong converse, improving by a constant factor the lower bound on $\mathsf{M}^{*}(\mathbf{A})$ obtained using Fano’s inequality by Candès and Davenport in [9]. Using the probabilistic method, [9] shows the existence of a packing set of well-separated vectors in $\Sigma_{k}$. To be specific:

Remark 5.1** (Packing set construction).**

[9, Lemma 2]** There exists a subset $\mathcal{X}\subseteq\Sigma_{k}$ of size $M:=\left\lvert\mathcal{X}\right\rvert=(n/k)^{k/4}$ whose elements ${\mathbf{u}}_{i}$ satisfy

$\left\lVert{\mathbf{u}}_{i}\right\rVert^{2}=1$. 2. 2.

$\left\lVert{\mathbf{u}}_{i}-{\mathbf{u}}_{j}\right\rVert^{2}\geq\frac{1}{2}$* for all $1\leq i,j\leq M$ such that $i\neq j$.* 3. 3.

$\left\lVert\frac{1}{M}\sum_{i=1}^{M}{\mathbf{u}}_{i}{\mathbf{u}}_{i}^{T}-\frac{1}{n}\mathbf{I}\right\rVert_{\rm{op}}\leq\beta/n$. Here $\beta$ is a constant that can be made arbitrarily small with growing $n$.

The set $\mathcal{X}$ gives a packing set ${\mathcal{P}}_{M,C/\sqrt{2}}:=\{\theta_{1},\ldots,\theta_{M}\}$ of codewords with minimum distance $\left\lVert\theta_{i}-\theta_{j}\right\rVert\geq\frac{C}{\sqrt{2}}$, simply by taking $\theta_{i}=C{\mathbf{u}}_{i}$, where the value of $C$ will be specified later.

In fact, we consider a subset of the packing set ${\mathcal{P}}_{M,C/\sqrt{2}}$, defined as follows:

Lemma 5.1**.**

Let $\delta_{M}\in[\frac{1}{M},1-\frac{1}{M}]$, where $M=(n/k)^{k/4}$. Then there exists a subset ${\mathcal{P}}_{M^{\prime},C/\sqrt{2}}\subseteq{\mathcal{P}}_{M,C/\sqrt{2}}$ such that $M^{\prime}:=\lceil\delta_{M}M\rceil$ and

[TABLE]

Proof.

We first bound the average over the packing set ${\mathcal{P}}_{M,C/\sqrt{2}}$, given by $\frac{1}{M}\sum_{i=1}^{M}\|\mathbf{A}\theta_{i}\|^{2}$. Using steps similar to those in [9, p.320], we have

[TABLE]

In the above chain, step $(a)$ holds because both $(\mathbf{A^{T}}\mathbf{A})$ and $\sum_{i=1}^{M}{\mathbf{u}}_{i}{\mathbf{u}}_{i}^{T}/M$ are positive semi-definite. Step $(b)$ is obtained using Property 3 of the packing set as defined in Remark 5.1.

We use the fact that if the average of $M$ non-negative numbers $c_{1}\leq c_{2}\ldots\leq c_{M}$ is $c$, then $c_{j}\leq\frac{c}{1-(j-1)/M}$, for $1\leq j\leq M$ (because otherwise the sum of the $(M-j+1)$ largest numbers will exceed $Mc$). The result then follows by picking $M^{\prime}$ elements of ${\mathcal{P}}_{M,C/\sqrt{2}}$ in increasing order of $\left\lVert\mathbf{A}\theta\right\rVert^{2}$, and calling this set ${\mathcal{P}}_{M^{\prime},C/\sqrt{2}}$. ∎

As we restrict attention to the subset ${\mathcal{P}}_{M^{\prime},C/\sqrt{2}}$ in the rest of this section, with mild abuse of notation, let us denote its elements by $\{\theta_{1},\ldots,\theta_{M^{\prime}}\}$. Also, let $\phi({\mathbf{y}};\boldsymbol{m},\boldsymbol{\Sigma})$ denote the normal density in $\mathbb{R}^{m}$ with mean vector $\boldsymbol{m}$ and covariance matrix $\boldsymbol{\Sigma}$. Then, with $\mu$ denoting the Lebesgue measure on $\mathcal{Y}$, from the measurement model (31), for any $\theta_{i}$ we have

[TABLE]

where the $r$th row of $\mathbf{A}$ is denoted by $\mathbf{A}^{T}_{r}\in\mathbb{R}^{n}$. Further, we choose

[TABLE]

and prove the following bound for the integral in (8).

Lemma 5.2**.**

Let $P_{{\mathbf{Y}}|\theta_{i}}$ and $Q_{{\mathbf{Y}}}$ given by (34) and (35), respectively. Then for any $\lambda>0$,

[TABLE]

Hence, the bracketed term in (8) can be bounded by

[TABLE]

Proof.

See Appendix E. ∎

Combining Lemma 37 with Theorem 1, we deduce the following lower bound.

Proposition 5.3**.**

For any $\lambda>0$, $\Delta\in(0,1)$, and $M=(n/k)^{k/4}$, we have

[TABLE]

where

[TABLE]

Therefore, for large $n$ we have

[TABLE]

Proof.

To apply Theorem 1, we equate the minimum distance $C/\sqrt{2}$ of the packing subset ${\mathcal{P}}_{M^{\prime},C/\sqrt{2}}^{\prime}$ to $2A\psi_{n}$. Taking $A=1$ gives $\psi_{n}=\frac{C}{2\sqrt{2}}$. Then, taking $w(t)=t^{2}$, we deduce that

[TABLE]

We can bound $\epsilon_{M}$ by using (37) of Lemma 37 in Theorem 1:

[TABLE]

Hence for any fixed $\lambda$ we obtain (38) and (39) by choosing

[TABLE]

with $\delta_{M}=1/\log M$ and $\Delta\in(0,1)$. To obtain (40), we recall from Remark 5.1 that $\beta$ can be chosen arbitrarily small as $n\to\infty$. Furthermore, $\lambda,\Delta$ can also be arranged to go [math] (at suitably slow rates) as $n\to\infty$. ∎

Remark 5.2**.**

The paper [9] uses Fano’s inequality to derive the following bound:

[TABLE]

Comparing with Proposition 5.3, we see that our result improves the bound by a factor close to 8 for large $n$.

Appendix A Proof of Lemma 10

For $(X,Y)\in\mathcal{X}\times\mathcal{Y}$ consider hypotheses $H_{0}:\;(X,Y)\sim Q$ and $H_{1}:\;(X,Y)\sim P$, where we assume that $P\ll Q$ so that the Radon-Nikodym derivative $\frac{dP}{dQ}$ exists.

The following lemma can be found in [34, Lemma 12.2] (also [2, eq. (102)]).

Lemma A.1**.**

For any randomized test $T$ to distinguish between the above hypotheses, and $\gamma>0$, we have

[TABLE]

We note that the maximum of the left hand side of (41) (over all all tests $T$) is the $E_{\gamma}$ divergence [35, 11]. We use (41) to complete the proof of Lemma 10:

Proof of Lemma 10.

As in [2, Theorem 26], let $\epsilon_{M}$ and $\epsilon^{\prime}_{M}$ denote the average error probabilities over channels $P_{{\mathbf{Y}}|\theta}$ and $Q_{{\mathbf{Y}}|\theta}=Q_{{\mathbf{Y}}}$, respectively, for a channel code with $M$ equiprobable codewords. Given $(\theta,{\mathbf{Y}})$, the result [2, Theorem 26] describes a (sub-optimal) hypothesis test based on the channel decoder to distinguish between $H_{0}:(\theta,{\mathbf{Y}})\sim Q_{\theta{\mathbf{Y}}}=\pi_{\theta}Q_{{\mathbf{Y}}}$ and $H_{1}:(\theta,{\mathbf{Y}})\sim P_{\theta{\mathbf{Y}}}=\pi_{\theta}P_{{\mathbf{Y}}|\theta}$. Let $T\in\{0,1\}$ denote the output of this test. It is shown in the proof of that theorem that the probability of Type I error, i.e., $Q[T=1]$ is $1-\epsilon^{\prime}_{M}$, and the probability of type II error, i.e., $P[T=0]=\epsilon_{M}$. Applying Lemma 41 to this hypothesis test yields that for any $\gamma>0$,

[TABLE]

We observe that when $Q_{{\mathbf{Y}}|\theta}=Q_{{\mathbf{Y}}}$, any channel decoder has average error probability $\epsilon^{\prime}_{M}=\frac{M-1}{M}$. The result in (10) follows by substituting this value for $\epsilon^{\prime}_{M}$ in (42). ∎

Appendix B Recovering Fano’s Inequality from Theorem 1

Here we show how to obtain Fano’s inequality from Theorem 1. We first establish a general converse result involving mutual information (equation (48)), and then obtain Fano’s inequality from it.

From the variational representation in (12), for any $\lambda,\gamma>0$ we have

[TABLE]

where H_{1+\lambda}(P\|Q):=\frac{1}{\lambda}\int\Big{(}\left(\frac{dP}{dQ}\right)^{1+\lambda}-1\Big{)}dQ is the Hellinger divergence of order $(1+\lambda)$ from distribution $P$ to distribution $Q$. We note from (9) that the Rényi and Hellinger divergences of order $(1+\lambda)$ are invertible functions of one another. We use the following bound [11, Theorem 8] for the Hellinger divergence:

[TABLE]

where

[TABLE]

We choose

[TABLE]

With this $Q_{{\mathbf{Y}}}$, we have that $t\leq M$ since ${\overline{P}}_{{\mathbf{Y}}}(\mathcal{A})\geq M^{-1}P_{{\mathbf{Y}}|\theta_{j}}(\mathcal{A})$ for all measurable sets $\mathcal{A}$. We also have

[TABLE]

Substituting in (44) and then in (43), we obtain

[TABLE]

Maximizing over $\gamma>0$ yields

[TABLE]

Taking logs, for any $\lambda>0$ we have

[TABLE]

where

[TABLE]

where the final inequality follows by direct comparison.

Hence, for a fixed $\lambda>0$ and $M\geq 3$, using (49) and $t\leq M$ in (48) gives

[TABLE]

Or,

[TABLE]

Finally, noting that $\lambda$ can be chosen arbitrarily small, choose $\lambda=1/(\log M)^{\alpha}$ for some $\alpha\in(0,1)$ in (50). We therefore have

[TABLE]

Using the expression for $I(\theta;\mathbf{Y})$ in (47) and rearranging, we get

[TABLE]

where $o(1)$ denotes a term that goes to zero with growing $M$. We have thus recovered Fano’s inequality in (5) to within $o(1)$ terms.

Appendix C Proof of Lemma 3.1

Proof of Lemma 3.1.

Since $Q_{{\mathbf{Y}}}$ is the uniform measure and each $\frac{dP_{{\mathbf{Y}}|\theta_{i}}}{dQ_{{\mathbf{Y}}}}$ corresponds to an $f_{{\boldsymbol{\tau}}}^{n}$, for each value of $i$, we can express the relevant integral as

[TABLE]

For any ${\boldsymbol{\tau}}$ we can express the bracketed term on the RHS of (51) as

[TABLE]

Here equality $(a)$ is obtained from (14) and (15) since $g_{j}(y)g_{k}(y)\equiv 0$ for $j\neq k$, and

[TABLE]

The result follows on substituting (52) into (51) and using $(1+x)^{n}\leq\exp(xn)$ for any $x\in{\mathbb{R}}$. ∎

Appendix D Proof of Lemma 4.1

Proof of Lemma 4.1.

We can express the key ratio on the LHS of (24) as

[TABLE]

We note that $dP_{{\mathbf{0}}}(V_{r}|U_{r})=P_{{\mathbf{0}}}(V_{r}|U_{r})d\nu(V_{r})$, where $\nu$ represents the counting measure on $\{0,1\}$. Then, using (53) we write the integral in (24) as

[TABLE]

where the inner integral $I_{n}$ can be written as

[TABLE]

where we use the fact that the Rényi divergence of order $(1+\lambda)$ between two Bernoulli random variables with parameters $\eta_{{\boldsymbol{\tau}}}(u_{n})$ and $\eta_{{\mathbf{0}}}(u_{n})$, respectively, is

[TABLE]

Recalling that $\beta_{m}=LHm^{-\alpha}$ and $u_{n}\in[0,1]^{d}$, let us denote the $d$th coordinate of $u_{n}$ by $u_{n,d}$. The construction of the hypercube class of functions ${\mathcal{F}}_{m}$ in [8, p.2350] ensures that for any ${\boldsymbol{\tau}},{\boldsymbol{\tau}^{\prime}}\in\{0,1\}^{m^{d-1}}$, the following properties hold.

[TABLE]

We now use the following bound on the Rényi divergence due to Verdú and Sason [12, Theorem 3] for $u_{n,d}\leq\beta_{m}$ and $\lambda\in[0,1]$:

[TABLE]

where $\delta:=\left\lvert\eta_{{\boldsymbol{\tau}}}(u_{n})-\eta_{{\mathbf{0}}}(u_{n})\right\rvert$ is the total variation distance between $P_{{\boldsymbol{\tau}}}(\cdot|u_{n})$ and $P_{0}(\cdot|u_{n})$. Using (57) for an upper bound on $\delta$, and (56) for a lower bound on the minimum of $P_{0}(\cdot|u_{n})$, we have from (58),

[TABLE]

Substituting this bound in (54) to bound $I_{n}$, we obtain using $1+x\leq e^{x}$ that

[TABLE]

The result follows by induction on $n$. ∎

Appendix E Proof of Lemma 37

Recall from (34) and (35) that we take $dP_{\theta_{i}}({\mathbf{y}}|\theta_{i})=\prod_{r=1}^{m}\phi(y_{r};\mathbf{A}^{T}_{r}\theta_{i},\sigma^{2})$ and $dQ_{{\mathbf{Y}}}({\mathbf{y}})=\prod_{r=1}^{m}\phi(y_{r};0,\sigma^{2})$.

Proof of Lemma 37.

For any $\lambda>0$, we have

[TABLE]

The equality in step $(a)$ is obtained by completing the square inside an exponential, and recognizing the remaining term as a multiple of a normal density.

To obtain (37), we use Lemma 5.1 to bound $\left\lVert\mathbf{A}\theta_{i}\right\rVert^{2}$ on the RHS of (36) for each $\theta_{i}\in{\mathcal{P}}_{M^{\prime},C/\sqrt{2}}$. ∎

Acknowledgements

The authors thank B. Nakiboğlu for pointing out the optimal choice of $Q_{{\mathbf{Y}}}$ given in Remark 2.3, and Igal Sason for helpful comments. They also thank the Alan Turing Institute for funding to attend the scoping workshop ‘Information-Theoretic Foundations and Algorithms for Large-Scale Data Inference’. RV would like to acknowledge support from a Marie Curie Career Integration Grant (Grant Number 631489).