An Introductory Guide to Fano's Inequality with Applications in Statistical Estimation
Jonathan Scarlett, Volkan Cevher
TL;DR
This paper surveys Fano's inequality and its variants, highlighting their importance in establishing impossibility results across various statistical estimation problems in information theory.
Contribution
It provides a comprehensive framework and key tools for applying Fano's inequality to diverse statistical estimation challenges.
Findings
Fano's inequality is versatile for impossibility proofs.
Applications include group testing, graphical models, and sparse regression.
The survey covers techniques and examples across multiple domains.
Abstract
Information theory plays an indispensable role in the development of algorithm-independent impossibility results, both for communication problems and for seemingly distinct areas such as statistics and machine learning. While numerous information-theoretic tools have been proposed for this purpose, the oldest one remains arguably the most versatile and widespread: Fano's inequality. In this chapter, we provide a survey of Fano's inequality and its variants in the context of statistical estimation, adopting a versatile framework that covers a wide range of specific problems. We present a variety of key tools and techniques used for establishing impossibility results via this approach, and provide representative examples covering group testing, graphical model selection, sparse linear regression, density estimation, and convex optimization.
| Sparse and low rank problems | Other estimation problems | ||
| Problem | References | Problem | References |
| Group testing Compressive sensing Sparse Fourier transform Principal component analysis Matrix completion | [2, 3] [4, 5] [6, 7] [8, 9] [10, 11] | Regression Density estimation Kernel methods Distributed estimation Local privacy | [12, 13] [14, 13] [15, 16] [17, 18] [19] |
| Sequential decision problems | Other learning problems | ||
| Problem | References | Problem | References |
| Convex optimization Active learning Multi-armed bandits Bayesian optimization Communication complexity | [20, 21] [22] [23] [24] [25] | Graph learning Ranking Classification Clustering Phylogeny | [26, 27] [28, 29] [30, 31] [32] [33] |
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
An Introductory Guide to Fano’s Inequality
with Applications in Statistical Estimation
Jonathan Scarlett1 and Volkan Cevher2
1 Department of Computer Science & Department of Mathematics, National University of Singapore, Singapore
2 Laboratory for Information and Inference Systems (LIONS), EPFL, Switzerland
[email protected], [email protected]
Abstract
Information theory plays an indispensable role in the development of algorithm-independent impossibility results, both for communication problems and for seemingly distinct areas such as statistics and machine learning. While numerous information-theoretic tools have been proposed for this purpose, the oldest one remains arguably the most versatile and widespread: Fano’s inequality. In this chapter, we provide a survey of Fano’s inequality and its variants in the context of statistical estimation, adopting a versatile framework that covers a wide range of specific problems. We present a variety of key tools and techniques used for establishing impossibility results via this approach, and provide representative examples covering group testing, graphical model selection, sparse linear regression, density estimation, and convex optimization.
Contents
-
A.3 Proof of Theorem 2 (Fano’s Inequality with Approximate Recovery)
-
A.7 Proof of Lemma 5 (Covering-Based Mutual Information Bound)
-
A.8 Omitted Details in Discrete Examples with Approximate Recovery
1 Introduction
The tremendous progress in large-scale statistical inference and learning in recent years has been spurred by both practical and theoretical advances, with strong interactions between the two: Algorithms that come with a priori performance guarantees are clearly desirable, if not crucial, in practical applications, and practical issues are indispensable in guiding the theoretical studies.
Complementary to performance bounds for specific algorithms, a key role is also played by algorithm-independent impossibility results, stating conditions under which one cannot hope to achieve a certain goal. Such results provide definitive benchmarks for practical methods, serve as certificates for near-optimality, and help guide the practical developments towards directions where the greatest improvements are possible.
Since its introduction in 1948, the field of information theory has continually provided such benefits for the problems of storing and transmitting data, and has accordingly shaped the design of practical communication systems. In addition, recent years have seen mounting evidence that the tools and methodology of information theory reach far beyond communication problems, and can provide similar benefits within the entire data processing pipeline.
While many information-theoretic tools have been proposed for establishing impossibility results, the oldest one remains arguably the most versatile and widespread: Fano’s inequality [1]. This fundamental inequality is not only ubiquitous in studies of communication, but has been applied extensively in statistical inference and learning problems; several examples are given in Table 1.
When applying Fano’s inequality to such problems, one typically encounters a number of distinct challenges compared to those found in communication problems. The goal of this chapter is to introduce the reader to some of the key tools and techniques, explain their interactions and connections, and provide several representative examples.
1.1 Overview of Techniques
Throughout the chapter, we consider the following statistical estimation framework, which captures a broad range of problems including the majority of those listed in Table 1:
- •
There exists an unknown parameter , known to lie in some set (e.g., a subset of ), that we would like to estimate.
- •
In the simplest case, the estimation algorithm has access to a set of samples drawn from some joint distribution parametrized by . More generally, the samples may be drawn from some joint distribution parametrized by , where are inputs that are either known in advance or selected by the algorithm itself.
- •
Given knowledge of , as well as if inputs are present, the algorithm forms an estimate of , with the goal of the two being “close” in the sense that some loss function is small. When referring to this step of the estimation algorithm, we will use the terms algorithm and decoder interchangeably.
We will initially use the following simple running example to exemplify some of the key concepts, and then turn to detailed applications in Sections 4 and 6.
Example 1**.**
(-sparse linear regression) A vector parameter is known to have at most one non-zero entry, and we are given linear samples of the form ,111Throughout the chapter, we interchange tuple-based notations such as , with vector/matrix notation such as , . where is a known input matrix, and is additive Gaussian noise. In other words, the -th sample is a noisy sample of , where is the transpose of the -th row of . The goal is to construct an estimate such that the squared distance is small.
This example is an extreme case of -sparse linear regression, in which has at most non-zero entries, i.e., at most columns of impact the output. The more general -sparse recovery problem will be considered in Section 6.1.
We seek to establish algorithm-independent impossibility results, henceforth referred to as converse bounds, in the form of lower bounds on the sample complexity, i.e., the number of samples required to achieve a certain average target loss. The following aspects of the problem significantly impact this goal, and their differences are highlighted throughout the chapter:
- •
Discrete vs. continuous: Depending on the application, the parameter set may be discrete or continuous. For instance, in the -sparse linear regression example, one may consider the case that is known to lie in a finite set , or one may consider the general estimation of a vector in the set
[TABLE]
where is the number of non-zeros in .
- •
Minimax vs. Bayesian: In the minimax setting, one seeks a decoder that attains a small loss for any given , whereas in the Bayesian setting, one considers the average performance under some prior distribution on . Hence, these two variations respectively consider the worst-case and average-case performance with respect to . We focus primarily on the minimax setting throughout the chapter, and further discuss Bayesian settings in Section 7.2.
- •
Choice of target goal: Naturally, the target goal can considerably impact the fundamental performance limits of an estimation problem. For instance, in discrete settings, it is common to consider exact recovery, requiring that (i.e., the 0-1 loss ), but it is also of interest to understand to what extent approximate recovery criteria make the problem easier.
- •
Non-adaptive vs. adaptive sampling: In settings consisting of an input as introduced above, one often distinguishes between the non-adaptive setting, in which is specified prior to observing any samples, and the adaptive setting, in which a given input can be designed based on the past inputs and samples . It is of significant interest to understand to what extent the additional freedom of adaptivity impacts the performance.
With these variations in mind, we proceed by outlining the main steps in obtaining converse bounds for statistical estimation via Fano’s inequality.
1.1.1 Step 1: Reduction to Multiple Hypothesis Testing
The multiple hypothesis testing problem is defined as follows: An index is drawn from a prior distribution , and a sequence of samples is drawn from a probability distribution parametrized by . The possible conditional distributions are known in advance, and the goal is to identify the index with high probability given the samples.
In Figure 1, we provide a general illustration of how an estimation problem can be reduced to multiple hypothesis testing, possibly with the added twist of including inputs . Supposing for the time being that we are in the minimax setting, the idea is to construct a hard subset of parameters that are difficult to distinguish given the samples. We then lower bound the worst-case performance by the average over this hard subset. As a concrete example, a good choice for the 1-sparse linear regression problem is to set and consider the set of vectors of the form
[TABLE]
where is a constant. Hence, the non-zero entry of has a given magnitude, which can be selected to our liking for the purpose of proving a converse.
We envision an index being drawn uniformly at random and used to select the corresponding parameter , and the estimation algorithm being run to produce an estimate . If the parameters are not too close and the algorithm successfully produces , then we should be able to infer the index from . This entire process can be viewed as a problem of multiple hypothesis testing, where the -th hypothesis is that the underlying parameter is (). With this reduction, we can deduce that if the algorithm performs well then the hypothesis test is successful; the contrapositive statement is then that if the hypothesis test cannot be successful, then the algorithm cannot perform well.
In the 1-sparse linear regression example, we find from (2) that distinct must satisfy . As a result, we immediately obtain from the triangle inequality that the following holds:
[TABLE]
In other words, if the algorithm yields , then can be identified as the index corresponding to the closest vector to . Thus, sufficiently accurate estimation implies success in identifying .
Discussion. Selecting the hard subset of parameters is often considered somewhat of an art. While the proofs of existing converse bounds may seem easy in hindsight when the hard subset is known, coming up with a suitable choice for a new problem usually requires some creativity and/or exploration. Despite this, there exist general approaches that have proved to been effective in a wide range problems, which we exemplify in Sections 4 and 6.
In general, selecting the hard subset requires balancing conflicting goals: Increasing so that the hypothesis test is more difficult, keeping the elements “close” so that they are difficult to distinguish, and keeping the elements “sufficiently distant” so that one can recover from . Typically, one of the following three approaches is adopted: (i) explicitly construct a set whose elements are known or believed to be difficult to distinguish; (ii) prove the existence of such a set using probabilistic arguments; or (iii) consider packing as many elements as possible into the entire space. We will provide examples of all three kinds.
In the Bayesian setting, is already random, so we cannot use the above-mentioned method of lower bounding the worst-case performance by the average. Nevertheless, if is discrete, we can still use the trivial reduction to form a multiple hypothesis testing problem with a possibly non-uniform prior. In the continuous Bayesian setting, one typically requires more advanced methods not covered in this chapter; we provide further discussion in Section 7.2.
1.1.2 Step 2: Application of Fano’s Inequality
Once a multiple hypothesis test is set up, Fano’s inequality provides a lower bound on its error probability in terms of the mutual information, which is one of the most fundamental information measures in information theory. The mutual information can often be explicitly characterized given the problem formulation, and a variety of useful properties are known for doing so, as outlined below.
We briefly state the standard form of Fano’s inequality Fano’s inequality for the case that is uniform on and is some estimate of :
[TABLE]
The intuition is as follows: The term represents the prior uncertainty (i.e., entropy) of , and the mutual information represents how much information reveals about . In order to have a small probability of error, we require that the information revealed is close to the prior uncertainty.
Beyond the standard form of Fano’s inequality (4), it is useful to consider other variants, including approximate recovery and conditional versions. These are the topic of Section 2, and we discuss other alternatives in Section 7.2.
1.1.3 Step 3: Bounding the Mutual Information
In order to make lower bounds such as (4) explicit, we need to upper bound the mutual information therein. This often consists of tedious yet routine calculations, but there are cases where it is highly non-trivial. The mutual information depends crucially on the choice of reduction in the first step.
The joint distribution of is decoder-dependent and usually very complicated, so to simplify matters, the typical first step is to apply an upper bound known as the data processing inequality. In the simplest case that there is no extra input to the sampling mechanism (i.e., is absent in Figure 1), this inequality takes the form under the Markov chain . Thus, we are left to answer the question of how much information the samples reveal about the index .
In Section 3, we introduce several useful tools for this purpose, including:
- •
Tensorization: If the samples are conditionally independent given , we have . Bounds of this type simplify the mutual information containing a set of observations to simpler terms containing only a single observation.
- •
KL divergence based bounds: Straightforward bounds on the mutual information reveal that if are close in terms of KL divergence, then the mutual information is small. Results of this type are useful, as the relevant KL divergences can often be evaluated exactly or tightly bounded.
In addition to these, we introduce variations for cases that the input is present in Figure 1, distinguishing between non-adaptive and adaptive sampling.
Toy example. To give a simple example of how this step is combined with the previous one, consider the case that we wish to identify one of hypotheses, with the -th hypothesis being that for some distribution on . That is, the observations are binary-valued. Starting with the above-mentioned bound , we simply write , which follows since takes one of at most values. Substitution into (4) yields , which means that achieving requires . This formalizes the intuitive fact that reliably identifying one of hypotheses requires roughly binary observations.
2 Fano’s Inequality and its Variants
In this section, we state various forms of Fano’s inequality that will form the basis for the results in the remainder of the chapter.
2.1 Standard Version
We begin with the most simple and widely-used form of Fano’s inequality. We use the generic notation for the discrete random variable in a multiple hypothesis test, and we write its estimate as . In typical applications, one has a Markov chain relation such as , where is the collection of samples; we will exploit this fact in Section 3, but for now, one can think of being randomly generated by any means given .
The two fundamental quantities appearing in Fano’s inequality are the conditional entropy , representing the uncertainty of given its estimate, and the error probability:
[TABLE]
Since , the conditional entropy is closely related to the mutual information, representing how much information reveals about .
Theorem 1**.**
(Fano’s inequality)* For any discrete random variables and on a common finite alphabet , we have*
[TABLE]
where is the binary entropy function. In particular, if is uniform on , we have
[TABLE]
or equivalently,
[TABLE]
Since the proof of Theorem 1 is widely accessible in standard references such as [34], we provide only an intuitive explanation of (6): To resolve the uncertainty in given , we can first ask whether the two are equal, which bears uncertainty . In the case that they differ, which only occurs a fraction of the time, the remaining uncertainty is at most \log\big{(}|\mathcal{V}|-1\big{)}.
Remark 1**.**
For uniform , we obtain (7) by upper bounding and in (6), and subtracting on both sides. While these additional bounds have a minimal impact for moderate to large values of , a notable case where one should use (6) is the binary setting, i.e., . In this case, (7) is meaningless due to the right-hand side being negative, whereas (6) yields the following for uniform :
[TABLE]
It follows that the error probability is lower bounded as
[TABLE]
where H_{2}^{-1}(\cdot)\in\big{[}0,\frac{1}{2}\big{]} is the inverse of on the domain \big{[}0,\frac{1}{2}\big{]}.
2.2 Approximate Recovery
The notion of error probability considered in Theorem 1 is that of exact recovery, insisting that . More generally, one can consider notions of approximate recovery, where one only requires to be “close” to in some sense. This is useful for at least two reasons:
- •
Exact recovery is often a highly stringent criterion in discrete statistical estimation problems, and it is of considerable interest to understand to what extent moving to approximate recovery makes the problem easier;
- •
When we reduce continuous estimation problems to the discrete setting (cf., Section 5), permitting approximate recovery will provide a useful additional degree of freedom.
We consider a general setup with a random variable , an estimate , and an error probability of the form
[TABLE]
for some real-valued function and threshold . In contrast to the exact recovery setting, there are interesting cases where and are defined on different alphabets, so we denote these by and , respectively.
One can interpret (11) as requiring to be within a “distance” of . However, need not be a true distance function, and need not even be symmetric nor take non-negative values. This definition of error probability in fact entails no loss of generality, since one can set and for an arbitrary set containing the pairs that are considered errors.
In the following, we make use of the quantities
[TABLE]
where
[TABLE]
counts the number of within a “distance” of .
Theorem 2**.**
(Fano’s inequality with approximate recovery)* For any random variables on the finite alphabets , we have*
[TABLE]
In particular, if is uniform on , then
[TABLE]
or equivalently
[TABLE]
The proof is similar to that of Theorem 1, and can be found in [35].
By setting and , we find that Theorem 2 recovers Theorem 1 as a special case. More generally, the bounds (15)–(16) resemble those for exact recovery in (7)–(8), but is replaced by . When , one can intuitively think of the approximate recovery setting as dividing the space into regions of size , and only requiring the correct region to be identified, thereby reducing the effective alphabet size to .
2.3 Conditional Version
When applying Fano’s inequality, it is often useful to condition on certain random events and random variables. The following theorem states a general variant of Theorem 1 with such conditioning. Conditional forms for the case of approximate recovery (Theorem 2) follow in an identical manner.
Theorem 3**.**
(Conditional Fano inequality)* For any discrete random variables and on a common alphabet , any discrete random variable on an alphabet , and any subset , the error probability satisfies*
[TABLE]
where . For possibly continuous , the same holds true with replaced by .
Proof.
We write , and lower bound the conditional error probability using Fano’s inequality (cf., Theorem 1) under the joint distribution of conditioned on . ∎
Remark 2**.**
Our main use of Theorem 3 will be to average over the input (cf., Figure 1) in the case that it is random and independent of . In such cases, by setting in (17) and letting contain all possible outcomes, we simply recover Theorem 1 with conditioning on in the conditional entropy and mutual information terms. The approximate recovery version, Theorem 2, extends in the same way. In Section 4, we will discuss more advanced applications of Theorem 3, including (i) genie arguments, in which some information about is revealed to the decoder, and (ii) typicality arguments, where we condition on falling in some high-probability set.
3 Mutual Information Bounds
We saw in Section 2 that the mutual information naturally arises from Fano’s inequality when is uniform. More generally, we have , so we can characterize the conditional entropy by characterizing both the entropy and the mutual information. In this section, we provide some of the main useful tools for upper bounding the mutual information. For brevity, we omit the proofs of standard results commonly found in information theory textbooks, or simple variations thereof.
Throughout the section, the random variables and are assumed to be discrete, whereas the other random variables involved, including the inputs and samples , may be continuous. Hence, notation such as may represent either a probability mass function (PMF) or a probability density function (PDF).
3.1 Data Processing Inequality
Recall the random variables , , , and in the multiple hypothesis testing reduction depicted in Figure 1. In nearly all cases, the first step in bounding a mutual information term such as is to upper bound it in terms of the samples , and possibly the inputs . By doing so, we remove the dependence on , and form a bound that is algorithm-independent.
The following lemma provides three variations along these lines. The three are all essentially equivalent, but are written separately since each will be more naturally suited to certain settings, as described below. Recall the terminology that forms a Markov chain if and are conditionally independent given , or equivalently, depends on only through .
Lemma 1**.**
(Data processing inequality)**
(i) If forms a Markov chain, then .
(ii) If forms a Markov chain conditioned on , then .
(iii) If forms a Markov chain, then .
We will use the first part when is absent or deterministic, the second part for random non-adaptive , and the third when the elements of can be chosen adaptively based on the past samples (cf. Section 1.1).
3.2 Tensorization
One of the most useful properties of mutual information is tensorization: Under suitable conditional independence assumptions, mutual information terms containing length- sequences (e.g., ) can be upper bounded by a sum of mutual information terms, the -th of which contains the corresponding entry of each associated vector (e.g., ). Thus, we can reduce a complicated mutual information term containing sequences to a sum of simpler terms containing individual elements. The following lemma provides some of the most common scenarios in which such tensorization can be performed.
Lemma 2**.**
(Tensorization of mutual information)* (i) If the entries of are conditionally independent given , then*
[TABLE]
(ii) If the entries of are conditionally independent given , and depends on only through , then
[TABLE]
(iii) If, in addition to the assumptions in part (ii), depends on only through for some deterministic function , then
[TABLE]
The proof is based on the sub-additivity of entropy, along with the conditional independence assumptions given. We will use the first part of the lemma when is absent or deterministic, and the second and third parts for random non-adaptive . When can be chosen adaptively based on the past samples (cf. Section 1.1), the following variant is used.
Lemma 3**.**
(Tensorization of mutual information for adaptive settings)* (i) If is a function of , and is conditionally independent of given , then*
[TABLE]
(ii) If, in addition to the assumptions in part (i), depends on only through for some deterministic function , then
[TABLE]
The proof is based on the chain rule for mutual information, i.e., , as well as suitable simplifications via the conditional independence assumptions.
Remark 3**.**
The mutual information bounds in Lemma 3 are analogous to those used in the problem of communication with feedback [34, Sec. 7.12]. A key difference is that in the latter setting, the channel input is a function of , with representing the message. In statistical estimation problems, the quantity being estimated is typically unknown to the decision-maker, so the input is only a function of
Remark 4**.**
Lemma 3 should be applied with care, since even if is uniform on some set a priori, it may not be uniform conditioned on . This is because in the adaptive setting, depends on , which in turn depends on .
3.3 KL Divergence Based Bounds
By definition, the mutual information is the KL divergence between the joint distribution and the product of marginals, , and can equivalently be viewed as a conditional divergence . Viewing the mutual information in this way leads to a variety of useful bounds in terms of related KL divergence quantities, as the following lemma shows.
Lemma 4**.**
(KL divergence based bounds)* Let , , and be the marginal distributions corresponding to a pair , where is discrete. For any auxiliary distribution , we have*
[TABLE]
and in addition,
[TABLE]
Proof.
We obtain (23) from the definition of mutual information, and (24) from the fact that \mathbb{E}\big{[}\log\frac{P_{Y|V}(Y|V)}{P_{Y}(Y)}\big{]}=\mathbb{E}\big{[}\log\frac{P_{Y|V}(Y|V)}{Q_{Y}(Y)}\big{]}-\mathbb{E}\big{[}\log\frac{P_{Y}(Y)}{Q_{Y}(Y)}\big{]}; the second term here is a KL divergence, and is therefore non-negative. We obtain (26) from (24) by noting that can be chosen to be any of the , and the remaining inequalities (25) and (27) are trivial. ∎
The upper bounds in (24)–(27) are closely related, and often essentially equivalent in the sense that they lead to very similar converse bounds. In the authors’ experience, it is usually slightly simpler to choose a suitable auxiliary distribution and apply (25), rather than bounding the pairwise divergences as in (27). Examples will be given in Sections 4 and 6.
Remark 5**.**
We have used the generic notation in Lemma 4, but in applications this may represent either the entire vector , or a single one of its entries . Hence, the lemma may be used to bound directly, or one may first apply tensorization and then use the lemma to bound each .
Remark 6**.**
Lemma 4 can also be used to bound conditional mutual information terms such as . Conditioned on any , we can upper bound using Lemma 4, with an auxiliary distribution that may depend on . For instance, doing this for (25) and then averaging over , we obtain for any that
[TABLE]
The bound (25) in Lemma 4 is useful when there exists a single auxiliary distribution that is “close” to each in KL divergence, i.e., D\big{(}P_{Y|V}(\cdot\,|\,v)\,\big{\|}\,Q_{Y}\big{)} is small. It is natural to extend this idea by introducing multiple auxiliary distributions, and only requiring that any one of them is close to a given . This can be viewed as “covering” the conditional distributions with “KL divergence balls”, and we will return to this viewpoint in Section 5.3.
Lemma 5**.**
(Mutual information bound via covering)* Under the setup of Lemma 4, suppose there exist distributions such that for all and some , it holds that*
[TABLE]
Then we have
[TABLE]
The proof is based on applying (24) with , and then lower bounding this summation over by the value achieving the minimum in (30). We observe that setting in Lemma 5 simply yields (25).
3.4 Relations Between KL Divergence and Other Measures
As evidenced above, the KL divergence plays a crucial role in applications of Fano’s inequality. In some cases, directly characterizing the KL divergence can still be difficult, and it is more convenient to bound it in terms of other divergences or distances. The following lemma gives a few simple examples of such relations; the reader is referred to [36] for a more thorough treatment.
Lemma 6**.**
(Relations between divergence measures)* Fix two distributions and , and consider the KL divergence D(P\|Q)=\mathbb{E}_{P}\big{[}\log\frac{P(Y)}{Q(Y)}\big{]}, total variation d_{\mathrm{TV}}(P,Q)=\frac{1}{2}\mathbb{E}_{Q}\big{[}\big{|}\frac{P(Y)}{Q(Y)}-1\big{|}\big{]}, squared Hellinger distance H^{2}(P,Q)=\mathbb{E}_{Q}\big{[}\big{(}\sqrt{\frac{P(Y)}{Q(Y)}}-1\big{)}^{2}\big{]}, and -divergence \chi^{2}(P\|Q)=\mathbb{E}_{Q}\big{[}\big{(}\frac{P(Y)}{Q(Y)}-1\big{)}^{2}\big{]}. We have:*
- •
(KL vs. TV)* , whereas if and are probability mass functions and each entry of is at least , then .*
- •
(Hellinger vs. TV)* ;*
- •
(KL vs. )* .*
4 Applications – Discrete Settings
In this section, we provide two examples of statistical estimation problems in which the quantity being estimated is discrete: group testing and graphical model selection. Our goal is not to treat these problems comprehensively, but rather, to study particular instances that permit a simple analysis while still illustrating the key ideas and tools introduced in the previous sections. We consider the high-dimensional setting, in which the underlying number of parameters being estimated is much higher than the number of measurements. To simplify the final results, we will often write them using the asymptotic notation for asymptotically vanishing terms, but non-asymptotic variants are easily inferred from the proofs.
4.1 Group Testing
The group testing problem consists of determining a small subset of “defective” items within a larger set of items based on a number of pooled tests. A given test contains some subset of the items, and the binary test outcome indicates, possibly in a noisy manner, whether or not at least one defective item was included in the test. This problem has a history in medical testing [37], and has regained significant attention following applications in communication protocols, pattern matching, database systems, and more.
In more detail, the setup is described as follows:
- •
In a population of items, there are unknown defective items. This defective set is denoted by , and is assumed to be uniform on the set of subsets having cardinality . Hence, in this example, we are in the Bayesian setting with a uniform prior. We focus on the sparse setting, in which , i.e., defective items are rare.
- •
There are tests specified by a test matrix : The -th entry of , denoted by , indicates whether item is included in test . We initially consider the non-adaptive setting, where is chosen in advance. We allow for this choice to be random; for instance, a common choice of random design is to let the entries of be i.i.d. Bernoulli random variables.
- •
To account for possible noise, we consider the following observation model:
[TABLE]
where for some \epsilon\in\big{[}0,\frac{1}{2}\big{)}, denotes modulo-2 addition, and is the “OR” operation. In the channel coding terminology, this corresponds to passing the noiseless test outcome through a binary symmetric channel. We assume that the noise variables are independent of each other and of , and we define the vector of test outcomes .
- •
Given and , a decoder forms an estimate of . We initially consider the exact recovery criterion, in which the error probability is given by
[TABLE]
where the probability with respect to , , and .
In the following subsections, we present several results and analysis techniques that are primarily drawn from [2, 3].
4.1.1 Exact Recovery with Non-Adaptive Testing
Under the exact recovery criterion (33), we have the following lower bound on the required number of tests. Recall that denotes the binary entropy function.
Theorem 4**.**
(Group testing with exact recovery)* Under the preceding noisy group testing setup, in order to achieve , it is necessary that*
[TABLE]
as , possibly with simultaneously.
Proof.
Since is discrete-valued, we can use the trivial reduction to multiple hypothesis testing with . Applying Fano’s inequality (cf., Theorem 1) with conditioning on (cf., Section 2.3), we obtain
[TABLE]
where we have also upper bounded using the data processing inequality (cf., second part of Lemma 1), which in turn uses the fact that conditioned on .
Let denote the hypothetical noiseless outcome. Since the noise variables are independent and depends on only through (cf., (32)), we can apply tensorization (cf., third part of Lemma 2) to obtain
[TABLE]
where (37) follows since is generated from according to a binary symmetric channel, which has capacity . Substituting (37) and {p\choose k}\geq\big{(}\frac{p}{k}\big{)}^{k} into (35) and rearranging, we obtain (34). ∎
Theorem 4 is known to be tight in terms of scaling laws whenever is fixed and , and perhaps more interestingly, tight including constant factors as under the scaling for sufficiently small . The matching achievability result in this regime can be proved using maximum-likelihood decoding [38]. However, achieving such a result using a computationally efficient decoder remains a challenging open problem.
4.1.2 Approximate Recovery with Non-Adaptive Testing
We now move to an approximate recovery criterion: The decoder outputs a list of cardinality , and we require that at least a fraction of the defective items appear in the list, for some . It follows that the error probability can be written as
[TABLE]
where , and . Notice that a higher value of means more non-defective items may be included in the list, whereas a higher value of means more defective items may be absent.
Theorem 5**.**
(Group testing with approximate recovery)* Under the preceding noisy group testing setup with list size , in order to achieve for some (not depending on ), it is necessary that*
[TABLE]
as , and simultaneously with .
Proof.
We apply the approximate recovery version of Fano’s inequality (cf., Theorem 2) with and as above. For any with cardinality , the number of with is given by , which follows by counting the number of ways to place defective items in , and the remaining defective items in the other entries. Hence, using Theorem 2 with conditioning on (cf., Section 2.3), and applying the data processing inequality (cf., second part of Lemma 1), we obtain
[TABLE]
By upper bounding the summation by times the maximum value, and performing some asymptotic simplifications via the assumption , we can simplify the logarithm to \big{(}k\log\frac{p}{L}\big{)}(1+o(1)) [39]. The theorem is then established by upper bounding the conditional mutual information using (37). ∎
Theorem 5 matches Theorem 4 up to the factor of and the replacement of by , suggesting that approximate recovery provides a minimal reduction in the number of tests even for moderate values of and . However, under approximate recovery, a near-matching achievability bound is known under the scaling for all , rather than only sufficiently small [38].
4.1.3 Adaptive Testing
Next, we discuss the adaptive testing setting, in which a given input vector , corresponding to a single row of , is allowed to depend on the previous inputs and outcomes, i.e., and . In fact, it turns out that Theorems 4 and 5 still apply in this setting. Establishing this simply requires making the following modifications to the above analysis:
- •
Apply the data processing inequality in the form of the third part of Lemma 1, yielding (35) and (40) with in place of ;
- •
Apply tensorization via Lemma 3 to deduce (36)–(37) with in place of .
In the regimes where Theorems 4 and/or 5 are known to have matching upper bounds with non-adaptive designs, we can clearly deduce that adaptivity provides no asymptotic gain. However, as with approximate recovery, adaptivity can significantly broaden the conditions under which matching achievability bounds are known, at least in the noiseless setting [40].
4.1.4 Discussion: General Noise Models
The preceding analysis can easily be extended to more general group testing models in which the observations are conditionally independent given . A broad class of such models can be written in the form , where denotes the number of defective items in the -th test. In such cases, the preceding results hold true more generally when is replaced by the capacity of the “channel” .
For certain models, we can obtain a better lower bound by applying a genie argument, along with the conditional form of Fano’s inequality in Theorem 3. Fix , and suppose that a uniformly random subset of cardinality is revealed to the decoder. This extra information can only make the group testing problem easier, so any converse bound for this modified setting remains valid for the original setting. Perhaps counter-intuitively, this idea can lead to a better final bound.
We only briefly outline the details of this more general analysis, and refer the interested reader to [3, 41]. Using Theorem 3 with , and applying the data processing inequality and tensorization, one can obtain
[TABLE]
where , and . The intuition is that we condition on since it is known via the genie, while the remaining information about is determined by . Once (41) is established, it only remains to simplify the mutual information terms; see [3, 41] for further details.
4.2 Graphical Model Selection
Graphical models provide compact representations of the conditional independence relations between random variables, and frequently arise in areas such as image processing, statistical physics, computational biology, and natural language processing. The fundamental problem of graphical model selection consists of recovering the graph structure given a number of independent samples from the underlying distribution.
Graphical model selection has been studied under several different families of joint distributions, and also several different graph classes. We focus our attention on the commonly-used Ising model with binary observations, and on a simple graph class known as forests, defined to contain the graphs having no cycles.
Formally, the setup is described as follows:
- •
We are given independent samples from a -dimensional joint distribution: for . This joint distribution is encoded by a graph , where is the vertex set, and is the edge set. We use the terminology vertex and node interchangeably. We assume that there are no edges from a vertex to itself, and that the edges are undirected: and are equivalent, and only count as one edge.
- •
We focus on the Ising model, in which the observations are binary-valued, and the joint distribution of a given sample, say , is
[TABLE]
where is a normalizing constant. Here is a parameter to the distribution dictating the edge strength; a higher value means it is more likely that for any given edge .
- •
We restrict the graph to be the set of all forests:
[TABLE]
where a cycle is defined to be a path of distinct edges leading back to the start node, e.g., . A special case of a forest is a tree, which is an acyclic graph for which a path exists between any two nodes. One can view any forest as being a disjoint union of trees, each defined on some subset of . See Figure 2 for an illustration.
- •
Let be the matrix whose -th row contains the entries of the -th sample. Given , a decoder forms an estimate of , or equivalently, an estimate of . We initially focus on the exact recovery criterion, in which the minimax error probability is given by
[TABLE]
where denotes probability when the true graph is , and the infimum is over all estimators.
To our knowledge, Fano’s inequality has not been applied previously in this exact setup; we do so using the general tools for Ising models given in [26, 27, 42, 43].
4.2.1 Exact Recovery
Under the exact recovery criterion, we have the following.
Theorem 6**.**
(Exact recovery of forest graphical models)* Under the preceding Ising graphical model selection setup with a given edge parameter , in order to achieve , it is necessary that*
[TABLE]
as .
Proof.
Recall from Section 1.1 that we can lower bound the worst-case error probability over by the average error probability over any subset of . This gives us an important degree of freedom in the reduction to multiple hypothesis testing, and corresponds to selecting a hard subset as described in Section 1.1.1. We refer to a given subset as a graph ensemble, and provide two choices that lead to the two terms in (45).
For any choice of , Fano’s inequality (Theorem 1) gives
[TABLE]
for uniform on , where we used by the data processing inequality and tensorization (cf. first parts of Lemmas 1 and 2).
Restricted ensemble 1: Let be the set of all trees. It is well-known from graph theory that the number of trees on nodes is [44]. Moreover, since is a length- binary sequence, we have . Hence, (46) yields , implying the first bound in (45).
Restricted ensemble 2: Let be the set of graphs containing a single edge, so that . We will upper bound the mutual information using (25) in Lemma 4, choosing the auxiliary distribution to be with being the empty graph. Thus, we need to bound for each .
We first give an upper bound on for any two graphs . We start with the trivial bound
[TABLE]
Recall the definition D(P\|Q)=\mathbb{E}_{P}\big{[}\log\frac{P(Y)}{Q(Y)}\big{]}, and consider the substitution of and according to (42), with different normalizing constants and . We see that when we sum the two terms in (47), the normalizing constants inside the logarithms cancel, and we are left with
[TABLE]
for and .
In the case that has a single edge (i.e., ) and is the empty graph, we can easily compute , and (48) simplifies to
[TABLE]
where is the unique edge in . Since and only take values in , we have , and letting have a single edge in (42) yields , and hence . Combining this with yields . Hence, using (49) along with (25) in Lemma 4, we obtain . Substitution into (46) (with ) yields the second bound in (45). ∎
Theorem 6 is known to be tight up to constant factors whenever [44, 45]: When is constant the lower bound becomes , whereas for asymptotically vanishing it simplifies to n=\Omega\big{(}\frac{1}{\lambda^{2}}\log p\big{)}.
4.2.2 Approximate Recovery
We consider the approximate recovery of with respect to the edit distance , which is the number of edge additions and removals needed to transform into or vice versa. Since any forest can have at most edges, it is natural to consider the case that an edit distance of up to is permitted, for some . Hence, the minimax risk is given by
[TABLE]
In this setting, we have the following.
Theorem 7**.**
(Approximate recovery of forest graphical models)* Under the preceding Ising graphical model selection setup with a given edge parameter and approximate recovery parameter \alpha\in\big{(}0,\frac{1}{2}\big{)} (with the latter not depending on ), in order to achieve , it is necessary that*
[TABLE]
as .
Proof.
For any , Theorem 2 provides the following analog of (46):
[TABLE]
for uniform on , where implicitly depends on . We again consider two restricted ensembles; the first is identical to the exact recovery setting, whereas the second is modified due to the fact that learning single-edge graphs with approximate recovery is trivial.
Restricted ensemble 1: Once again, let be the set of all trees. We have already established and for this ensemble, so it only remains to characterize .
While the decoder may output a graph not lying in , we can assume without loss of generality that is always selected such that for some ; otherwise, an error would be guaranteed. As a result, for any , and any such that , we have from the triangle inequality that , which implies that
[TABLE]
Now observe that since all graphs in have exactly edges, transforming to requires removing edges and adding different edges, for some . Hence, we have
[TABLE]
By upper bounding the summation by times the maximum, and performing some asymptotic simplifications, we can show that \log N_{\mathrm{\max}}(\alpha p)\leq\big{(}\alpha p\log p\big{)}(1+o(1)). Substituting into (52) and recalling that and , we obtain the first bound in (51).
Restricted ensemble 2a: Let be the set of all graphs on nodes containing exactly isolated edges; if is an odd number, the same analysis applies with an arbitrary single node ignored. We proceed by characterizing , , and . The number of graphs in the ensemble is , and Stirling’s approximation yields \log|\mathcal{G}_{2a}|\geq\big{(}p\log p\big{)}(1+o(1)).
Since the KL divergence is additive for product distributions, and we established in the exact recovery case that the KL divergence between the distributions of a single-edge graph and an empty graph is at most , we deduce that for any , where is the empty graph. We therefore obtain from Lemma 4 that .
A similar argument to that of Ensemble 1 yields , in analogy with (54). This again simplifies to N_{\mathrm{\max}}(\alpha p)\leq\big{(}\alpha p\log p\big{)}(1+o(1)), and having established \log|\mathcal{G}_{2a}|\geq\big{(}p\log p\big{)}(1+o(1)) and , substitution into (52) yields the second bound in (51). ∎
The bound in Theorem 7 matches that of Theorem 6 up to a multiplicative factor of , thus suggesting that approximate recovery does not significantly help in reducing the required number of samples, at least in the minimax sense, for the Ising model and forest graph class.
4.2.3 Adaptive Sampling
We now return to the exact recovery setting, and consider a modification in which we have an added degree of freedom in the form of adaptive sampling:
- •
The algorithm proceeds in rounds; in round , the algorithm queries a subset of the nodes indexed by , and the corresponding sample is generated as follows:
- –
The joint distribution of the entries of , corresponding to the entries where is one, coincides with the corresponding marginal distribution of , with independence between rounds;
- –
The values of the entries of , corresponding to the entries where is zero, are given by , a symbol indicating that the node was not observed.
We allow to be selected based on the past queries and samples, namely, and .
- •
Let denote the number of ones in , i.e., the number of nodes observed in round . While we allow the total number of rounds to vary, we restrict the algorithm to output an estimate after observing at most nodes. This quantity is related to in the non-adaptive setting according to , since in the non-adaptive setting we always observe all nodes in each sample.
- •
The minimax risk is given by
[TABLE]
where the infimum is over all adaptive algorithms that observe at most nodes in total.
Theorem 8**.**
(Adaptive sampling for forest graphical models)* Under the preceding Ising graphical model selection problem with adaptive sampling and a given parameter , in order to achieve , it is necessary that*
[TABLE]
as .
Proof.
We prove the result using Ensemble 1 and Ensemble 2a above. We let denote the number of rounds; while this quantity is allowed to vary, we can assume without loss of generality that by adding or removing rounds where no nodes are queried. For any subset , applying Fano’s inequality (cf., Theorem 1) and tensorization (cf., first part of Theorem 3) yields
[TABLE]
where is uniform on .
Restricted ensemble 1: We again let be the set of all trees, for which we know that . Since the entries of differing from are binary, and those equaling are deterministic given , we have . Averaging over and summing over yields , and substitution into (57) yields the first bound in (56).
Restricted ensemble 2a: We again use the above-defined ensemble of graphs with isolated edges, for which we know that |\mathcal{G}_{2a}|\geq\big{(}p\log p\big{)}(1+o(1)). In this case, when we observe nodes, the sub-graph corresponding to these observed nodes has at most edges, all of which are isolated. Hence, using Lemma 4, the above-established fact that the KL divergence from a single-edge graph to the empty graph is at most , and the additivity of KL divergence for product distributions, we deduce that . Averaging over and summing over yields , and substitution into (57) yields the second bound in (56). ∎
The threshold in Theorem 8 matches that of Theorem 6, and in fact, a similar analysis under approximate recovery also recovers the threshold in Theorem 7. This suggests that adaptivity is of limited help in the minimax sense for the Ising model and forest graph class. There are, however, other instances of graphical model selection where adaptivity provably helps [46, 43].
4.2.4 Discussion: Other Graph Classes
Degree and edge constraints: While the class is a relatively easy class to handle, similar techniques have also been used for more difficult classes, notably including those that place restrictions on the maximal degree and/or the number of edges . Ensembles 2 and 2a above can again be used, and the resulting bounds are tight in certain scaling regimes where , but loose in other regimes due to their lack of dependence on and . To obtain bounds with such a dependence, alternative ensembles have been proposed consisting of sub-graphs with highly correlated nodes [26, 27, 42].
For instance, suppose that a group of nodes has all possible edges connected except one. Unless or the edge strength are small, the high connectivity makes the nodes very highly correlated, and the sub-graph is difficult to distinguish from a fully-connected sub-graph. This is in contrast with Ensembles 2 and 2a above, whose graphs are difficult to distinguish from the empty graph.
Bayesian setting: Beyond minimax estimation, it is also of interest to understand the fundamental limits of random graphs. A particularly prominent example is the Erdös-Rényi random graph, in which each edge is independently included with some probability . This is a case where the conditional form of Fano’s inequality has proved useful; specifically, one can apply Theorem 3 with , and equal to the following typical set of graphs:
[TABLE]
where is a constant. Standard properties of typical sets [34] yield that , |\mathcal{T}|=e^{\big{(}H_{2}(q){p\choose 2}\big{)}(1+O(\epsilon))}, and H(V|V\in\mathcal{T})=\big{(}H_{2}(q){p\choose 2}\big{)}(1+O(\epsilon)) whenever , and once these facts are established, Theorem 3 yields the following following necessary condition for :
[TABLE]
For instance, in the case that q=O\big{(}\frac{1}{p}\big{)} (i.e., there are edges on average), we have H_{2}(q)=\Theta\big{(}\frac{\log p}{p}\big{)}, and we find that samples are necessary. This scaling is tight when is constant [45], whereas improved bounds for other scalings can be found in [27].
5 From Discrete to Continuous
Thus far, we have focused on using Fano’s inequality to provide converse bounds for the estimation of discrete quantities. In many, if not most, statistical applications, one is instead interested in estimating continuous quantities; examples include linear regression, covariance estimation, density estimation, and so on. It turns out that the discrete form of Fano’s inequality is still broadly applicable in such settings. The idea, as outlined in Section 1, is to choose a finite subset that still captures the inherent difficulty in the problem. In this section, we present several tools used for this purpose.
5.1 Minimax Estimation Setup
Recall the setup described in Section 1.1: A parameter is known to lie in some subset of a continuous domain (e.g., ), the samples are drawn from a joint distribution , an estimate is formed, and the loss incurred is . For clarity of exposition, we focus primarily on the case that there is no input, i.e., in Figure 1 is absent or deterministic. However, the main results (cf., Theorems 9 and 10 below) extend to settings with inputs as described in Section 1.1; the mutual information is replaced by in the non-adaptive setting, or in the adaptive setting.
In continuous settings, the reduction to multiple hypothesis testing (cf., Figure 1) requires that the loss function is sufficiently well-behaved. We focus on a widely-considered class of functions that can be written as
[TABLE]
where is a metric, and is an increasing function from to . For instance, the squared- loss clearly takes this form.
We focus on the minimax setting, defining the minimax risk as follows:
[TABLE]
where the infimum is over all estimators , and denotes expectation when the underlying parameter is . We subsequently define analogously.
5.2 Reduction to the Discrete Case
We present two related approaches to reducing the continuous estimation problem to a discrete one. The first, based on the standard form of Fano’s inequality in Theorem 1, was discovered much earlier [12], and accordingly, it has been used in a much wider range of applications. However, the second approach, based on the approximate recovery version of Fano’s inequality in Theorem 2, has recently been shown to provide added flexibility in the reduction [35].
5.2.1 Reduction with Exact Recovery
As we discussed in Section 1, we seek to reduce the continuous problem to multiple hypothesis testing in such a way that successful minimax estimation implies success in the hypothesis test with high probability. To this end, we choose a hard subset , for which the elements are sufficiently well-separated so that the index can be identified from the estimate (cf., Figure 1). This is formalized in the proof of the following result.
Theorem 9**.**
(Minimax bound via reduction to exact recovery)* Under the preceding minimax estimation setup, fix , and let be a finite subset of such that*
[TABLE]
Then, we have
[TABLE]
where is uniform on , and the mutual information is with respect to . Moreover, in the special case , we have
[TABLE]
where is the inverse binary entropy function.
Proof.
As illustrated in Figure 1, the idea is to reduce the estimation problem to a multiple hypothesis testing problem. As an initial step, we note from Markov’s inequality that, for any ,
[TABLE]
where (66) uses (60) and the assumption that is increasing.
Suppose that a random index is drawn uniformly from , the samples are drawn from the distribution corresponding to , and the estimator is applied to produce . Let correspond to the closest according to the metric , i.e., . Using the triangle inequality and the assumption (62), if then we must have ; hence,
[TABLE]
where is a shorthand for .
With the above tools in place, we proceed as follows:
[TABLE]
where (68) follows by maximizing over a smaller set, (69) follows from (67), (70) lower bounds the maximum by the average, and (71) follows from Fano’s inequality (cf., Theorem 1) and the fact that by the data processing inequality (cf., Lemma 1).
The proof of (63) is concluded by substituting (71) into (66) with , and taking the infimum over all estimators . For , we obtain (64) in the same way upon replacing (71) by the version of Fano’s inequality for given in Remark 1. ∎
We return to this result in Section 5.3, where we introduce and compare some of the most widely-used approaches to choosing the set and bounding the mutual information.
5.2.2 Reduction with Approximate Recovery
The following generalization of Theorem 9, based on Fano’s inequality with approximate recovery (cf., Theorem 2), provides added flexibility in the reduction. An example comparing the two approaches will be given in Section 6 for the sparse linear regression problem.
Theorem 10**.**
(Minimax bound via reduction to approximate recovery)* Under the preceding minimax estimation setup, fix , , a finite set of cardinality , and an arbitrary real-valued function on , and let be a finite subset of such that*
[TABLE]
Then we have for any that
[TABLE]
where is uniform on , the mutual information is with respect to , and .
The proof is analogous to that of Theorem 9, and can be found in [35].
5.3 Local vs. Global Approaches
Here we highlight two distinct approaches to applying the reduction to exact recovery as per Theorem 9, termed the local and global approaches. We do not make such a distinction for the approximate recovery variant in Theorem 10, since we are not aware of a global approach being used previously for this variant.
Local approach. The most common approach to applying Theorem 9 is to construct a set of elements that are “close” in KL divergence. Specifically, upper bounding the mutual information via Lemma 4 (with the vector playing the role of therein), one can weaken (63) as follows.
Corollary 1**.**
(Local approach to minimax estimation)* Under the setup of Theorem 9 with a given set satisfying (62), it holds for any auxiliary distribution that*
[TABLE]
Moreover, the same bound holds true when is replaced by any of , , or .
Attaining a good bound in (74) requires choosing to trade off two competing objectives: (i) A larger value of means that more hypotheses need to be distinguished; and (ii) A smaller value of means that the hypotheses are more similar. Generally speaking, there is no single best approach to optimizing this trade-off, and the size and structure of the set can vary significantly from problem to problem. Moreover, the construction need not be explicit; one can instead use probabilistic arguments to prove the existence of a set satisfying the desired properties. Examples are given in Section 6. Naturally, an analog of Corollary 1 holds for as per Theorem 9, and a counterpart for approximate recovery holds as per Theorem 10.
We briefly mention that Corollary 1 has interesting connections with the popular Assouad method from the statistics literature, as detailed in [47]. In addition, the counterpart of Corollary 1 with (using (10) in its proof) is similarly related to an analogous technique known as Le Cam’s method.
Global approach. An alternative approach to applying Theorem 9 is the global approach, which performs the following: (i) Construct a subset of with as many elements as possible subject to the assumption (62); (ii) Construct a set that covers , in the sense of Lemma 5, with as few elements as possible. The following definitions formalize the notions of forming “as many” and “as few” elements as possible. We write these in terms of a general real-valued function that need not be a metric.
Definition 1**.**
A set is said to be an -packing set of with respect to a measure if for all with . The -packing number is defined to be the maximum cardinality of any -packing.
Definition 2**.**
A set is said to be an -covering set of with respect to if, for any , there exists some such that . The -covering number is defined to be the minimum cardinality of any -covering.
Observe that assumption (62) of Theorem 9 precisely states that is an -packing set, though the result is often applied with far smaller than the -packing number. The logarithm of the covering number is often referred to as the metric entropy.
The notions of packing and covering are illustrated in Figure 3. We do not explore the properties of packing and covering numbers in detail in this chapter; the interested reader is referred to [48, 49] for a more detailed treatment. We briefly state the following useful property, showing that the two definitions are closely related in the case that is a metric.
Lemma 7**.**
(Packing vs. covering numbers)* If is a metric, then .*
We now show how to use Theorem 9 to construct a lower bound on the minimax risk in terms of certain packing and covering numbers. For the packing number, we will directly consider the metric used in Theorem 9. On the other hand, for the covering number, we consider the density associated with each , and use the associated KL divergence measure:
[TABLE]
Corollary 2**.**
(Global approach to minimax estimation)* Under the minimax estimation setup of Section 5.1, we have for any and that*
[TABLE]
In particular, if is the -fold product of some single-measurement distribution for each , then we have for any and that
[TABLE]
where with .
Proof.
Since Theorem 9 holds for any packing set, it holds for the maximal packing set. Moreover, using Lemma 5, we have in (63), since covering the entire space is certainly enough to cover the elements in the packing set. Combining these, we obtain the first part of the corollary. The second part follows directly from the first part by choosing and noting that the KL divergence is additive for product distributions. ∎
Corollary 2 has been used as the starting point to derive minimax lower bounds for a wide range of problems [13]; see Section 6 for an example. It has been observed that the global approach is mainly useful for infinite-dimensional problems such as density estimation and non-parametric regression, with the local approach typically being superior for finite-dimensional problems such as vector or matrix estimation.
5.4 Beyond Estimation – Fano’s Inequality for Optimization
While the minimax estimation framework captures a diverse range of problems of interest, there are also interesting problems that it does not capture. A notable example, which we consider in this section, is stochastic optimization. We provide a brief treatment, and refer the reader to [20] for further details and results.
We consider the following setup:
- •
We seek to minimize an unknown function on some input domain , i.e., to find a point such that is as low as possible.
- •
The algorithm proceeds in iterations: At the -th iteration, a point is queried, and an oracle returns a sample depending on the function, e.g., a noisy function value, a noisy gradient, or a tuple containing both. The selected point can depend on the past queries and samples.
- •
After iteratively sampling points, the optimization algorithm returns a final point , and the loss incurred is , i.e., the gap to the optimal function value.
- •
For a given class of functions , the minimax risk is given by
[TABLE]
where the infimum is over all optimization algorithms that iteratively query the function times and return a final point as above, and denotes expectation when the underlying function is .
In the following, we let and denote the queried locations and samples across the rounds.
Theorem 11**.**
(Minimax bound for noisy optimization)* Fix , and let be a finite subset of such that for each , we have for at most one value of . Then we have*
[TABLE]
where is uniform on , and the mutual information is with respect to . Moreover, in the special case , we have
[TABLE]
where is the inverse binary entropy function.
Proof.
By Markov’s inequality, we have
[TABLE]
Suppose that a random index is drawn uniformly from , and the triplet is generated by running the optimization algorithm on . Given , let index the function among with the lowest corresponding value: .
By the assumption that any satisfies for at most one of the functions, we find that the condition implies . Hence, we have
[TABLE]
The remainder of the proof follows (68)–(71) in the proof of Theorem 9: We lower bound the minimax risk \sup_{f\in\mathcal{F}}\mathbb{P}_{f}\big{[}\ell_{f}(\hat{X})\geq\epsilon\big{]} by the average over , and apply Fano’s inequality (cf., Theorem 1 and Remark 1) and the data processing inequality (cf., third part of Lemma 3). ∎
Remark 7**.**
Theorem 10 is based on reducing the optimization problem to a multiple hypothesis testing problem with exact recovery. One can derive an analogous result reducing to approximate recovery, but we are unaware of any works making use of such a result for optimization.
6 Applications – Continuous Settings
In this section, we present three applications of the tools introduced in Section 5: sparse linear regression, density estimation, and convex optimization. Similarly to the discrete case, our examples are chosen to permit a relatively simple analysis, while still effectively exemplifying the key concepts and tools.
6.1 Sparse Linear Regression
In this example, we extend the -sparse linear regression example of Section 1.1 to the more general scenario of -sparsity. The setup is described as follows:
- •
We wish to estimate a high-dimensional vector that is -sparse: , where is the number of non-zero entries in .
- •
The vector of measurements is given by , where is a known deterministic matrix, and is additive Gaussian noise.
- •
Given knowledge of and , an estimate is formed, and the loss is given by the squared -error, , corresponding to (60) with and . Overloading the general notation , we write the minimax risk as
[TABLE]
where denotes expectation when the underlying vector is .
6.1.1 Minimax Bound
The lower bound on the minimax risk is formally stated as follows. To simplify the analysis slightly, we state the result in an asymptotic form for the sparse regime ; with only minor changes, one can attain a non-asymptotic variant attaining the same scaling laws for more general choices of [35].
Theorem 12**.**
(Sparse linear regression)* Under the preceding sparse linear regression problem with and a fixed regression matrix , we have*
[TABLE]
as . In particular, under the constraint for some , achieving requires .
Proof.
We present a simple proof based on a reduction to approximate recovery (cf., Theorem 10). In Section 6.1.2, we discuss an alternative proof based on a reduction to exact recovery (cf., Theorem 9).
We define the set
[TABLE]
and to each , we associate a vector for some . Letting denote the Hamming distance, we have the following properties:
- •
For , if , then ;
- •
The cardinality of is , yielding ;
- •
The quantity in Theorem 10 is the maximum possible number of such that for a fixed . Setting , a simple counting argument gives N_{\mathrm{\max}}(t)\leq\sum_{j=0}^{\lceil k/2\rceil}2^{j}{p\choose j}\leq\big{(}\lceil\frac{k}{2}\rceil+1\big{)}\cdot 2^{\lceil k/2\rceil}\cdot{p\choose\lceil k/2\rceil}, which simplifies to \log N_{\mathrm{\max}}(t)\leq\big{(}\frac{k}{2}\log\frac{p}{k}\big{)}(1+o(1)) due to the assumption .
From these observations, applying Theorem 10 with and yields
[TABLE]
Note that we do not condition on in the mutual information, since we have assumed that is deterministic.
To bound the mutual information, we first apply tensorization (cf., first part of Lemma 2) to obtain , and then bound each using equation (24) in Lemma 4. We let be the density function, and we let denote the density function of , where is the transpose of the -th row of . Since the KL divergence between the and density functions is , we have . As a result, Lemma 4 yields I(V;Y_{i})\leq\frac{1}{|\mathcal{V}|}\sum_{v}D(P_{v,i}\|Q_{Y})=\frac{1}{2\sigma^{2}}\mathbb{E}\big{[}|X_{i}^{T}\theta_{V}|^{2}\big{]} for uniform . Summing over and recalling that , we deduce that
[TABLE]
From the choice of in (85), we can easily compute , which implies that . Substitution into (87) yields , and we conclude from (86) that
[TABLE]
The proof is concluded by setting , which is chosen to make the bracketed term tend to . ∎
Up to constant factors, the lower bound in Theorem 12 cannot be improved without additional knowledge of beyond its Frobenius norm [5]. For instance, in the case that has i.i.d. Gaussian entries, a matching upper bound holds with high probability under maximum-likelihood decoding.
6.1.2 Alternative Proof: Reduction with Exact Recovery
In contrast to the proof given above (adapted from [35]), the first known proof of Theorem 12 was based on packing with exact recovery (cf., Theorem 9) [5]. For the sake of comparison, we briefly outline this alternative approach, which turns out to be more complicated.
The main step is to prove the existence of a set satisfying the following properties:
- •
The number of elements satisfies M=\Omega\big{(}k\log\frac{p}{k}\big{)};
- •
Each element is -sparse with non-zero entries equal to ;
- •
The elements are well-separated in the sense that for ;
- •
The empirical covariance matrix is close to a scaled identity matrix in the following sense: \big{\|}\frac{1}{M}\sum_{v=1}^{M}\theta_{v}\theta_{v}^{T}-\frac{k}{p}\cdot\mathbf{I}_{p}\|_{2\to 2}=o\big{(}\frac{k}{p}\big{)}, where denotes the -operator norm, i.e., the largest singular value.
Once this is established, the proof proceeds along the same lines as the proof we gave above, scaling the vectors down by some and using Theorem 9 in place of Theorem 10.
The existence of the packing set is proved via a probabilistic argument: If one generates \Omega\big{(}k\log\frac{p}{k}\big{)} uniformly random -sparse sequences with non-zero entries equaling , then these will satisfy the remaining two properties with positive probability. While it is straightforward to establish the condition of being well-separated, the proof of the condition on the empirical covariance matrix requires a careful application of the non-elementary matrix Bernstein inequality.
Overall, while the two approaches yield the same result up to constant factors in this example, the approach based on approximate recovery is entirely elementary and avoids the preceding difficulties.
6.2 Density Estimation
In this subsection, we consider the problem of estimating an entire probability density function given samples from its distribution, commonly known as density estimation. We consider a non-parametric view, meaning that the density does not take any specific parametric form. As a result, the problem is inherently infinite-dimensional, and lends itself to the global packing and covering approach introduced in Section 5.3.
While many classes of density functions have been considered in the literature [13], we focus our attention on a specific setting for clarity of exposition:
- •
The density function that we seek to estimate is defined on the domain , i.e., for all , and .
- •
We assume that satisfies the following conditions:
[TABLE]
for some and , where the total variation (TV) norm is defined as \|f\|_{\mathrm{TV}}=\sup_{L}\sup_{0\leq x_{1}\leq\dotsc\leq x_{L}\leq 1}\sum_{l=2}^{L}\big{(}f(x_{l})-f(x_{l-1})\big{)}. The set of all density functions satisfying these constraints is denoted by .
- •
Given independent samples from , an estimate is formed, and the loss is given by . Hence, the minimax risk is given by
[TABLE]
where denotes expectation when the underlying density is .
6.2.1 Minimax Bound
The minimax lower bound is given is follows.
Theorem 13**.**
(Density estimation)* Consider the preceding density estimation setup with some and not depending on . There exists a constant (depending on and ) such that in order to achieve , it is necessary that*
[TABLE]
when is sufficiently small. In other words, \mathcal{M}_{n}(\eta,\Gamma)=\Omega\big{(}n^{-2/3}\big{)}.
Proof.
We specialize the general analysis of [13] to the class . Recalling the packing and covering numbers from Definitions 1 and 2, we adopt the shorthand notation with , and similarly . We first show that (cf., Corollary 2) can be upper bounded in terms of , which will lead to a minimax lower bound that depends only on the packing number . For , we have
[TABLE]
where (92) follows since the KL divergence is upper bounded by the -divergence (cf., Lemma 6), and (93) follows from the assumption that the density is lower bounded by . From the definition of in Corollary 2, we deduce the following for any :
[TABLE]
where the first inequality holds because any -covering the the -norm is also a -covering in the KL divergence due to (94), and the second inequality follows from Lemma 7.
Combining (95) with Corollary 2 and the choice gives
[TABLE]
We now apply the following bounds on the packing number of , which we state from [13] without proof:
[TABLE]
for some constants and sufficiently small . It follows that
[TABLE]
The remainder of the proof amounts to choosing and to balance the terms appearing in this expression.
First, choosing to equate the terms and leads to \epsilon_{\mathrm{c}}=\big{(}\frac{c^{\prime}}{n}\big{)}^{2/3} with , yielding \frac{\overline{c}\cdot(\eta\epsilon_{\mathrm{c}})^{-1/2}+n\epsilon_{\mathrm{c}}+\log 2}{\underline{c}\cdot\epsilon_{\mathrm{p}}^{-1}}=\frac{2n\big{(}\frac{c^{\prime}}{n}\big{)}^{2/3}+\log 2}{\overline{c}\cdot\epsilon_{\mathrm{p}}^{-1}}. Next, choosing to make this fraction equal to yields \epsilon_{\mathrm{p}}^{-1}=\frac{2}{\overline{c}}\big{(}2(c^{\prime})^{2/3}n^{1/3}+\log 2\big{)}, which means that for suitable and sufficiently large . Finally, since we made the fraction equal to , (98) yields . Setting and solving for yields the desired result. ∎
The scaling given in Theorem 13 cannot be improved; a matching upper bound is given in [13], and can be achieved even when .
6.3 Convex Optimization
In our final example, we consider the optimization setting introduced in Section 5.4. We provide an example that is rather simple, yet has interesting features not present in the previous examples: (i) an example departing from estimation; (ii) a continuous example with adaptivity; and (iii) a case where Fano’s inequality with is used.
We consider the following special case of the general setup of Section 5.4:
- •
We let be the set of differentiable and strongly convex functions on , with strong convexity parameter equal to one:
[TABLE]
The analysis that we present can easily be extended to functions on an arbitrary closed interval with an arbitrary strong convexity parameter.
- •
When we query a point , we observe a noisy sample of the function value and its gradient:
[TABLE]
where and are independent random variables, for some . This is commonly referred to as the noisy first-order oracle.
6.3.1 Minimax Bound
The following theorem lower bounds the number of queries required to achieve -optimality. The proof is taken from [20] with only minor modifications.
Theorem 14**.**
(Stochastic optimization of strongly convex functions)* Under the preceding convex optimization setting with noisy first-order oracle information, in order to achieve , it is necessary that*
[TABLE]
when is sufficiently small.
Proof.
We construct a set of two functions satisfying the assumptions of Theorem 11. Specifically, we fix such that , define and , and set
[TABLE]
These functions are illustrated in Figure 4.
Since \epsilon^{\prime}\in\big{(}0,\frac{1}{8}\big{)}, both and lie in , and hence . Moreover, a direct evaluation reveals that f_{1}(x)+f_{2}(x)=\big{(}x-\frac{1}{2}\big{)}^{2}+2\epsilon^{\prime}>2\epsilon, which implies that any -optimal point for one function cannot be -optimal for the other function. This is the condition needed to apply Theorem 11, yielding from (80) that
[TABLE]
To bound the mutual information, we first apply tensorization (cf., first part of Lemma 3) to obtain . We proceed by bounding for any given . Fix , let and be the density functions of the noisy samples of and , and let and be defined similarly for f_{0}(x)=\frac{1}{2}\big{(}x-\frac{1}{2}\big{)}^{2}. We have
[TABLE]
where (104) holds since the KL divergence is additive for product distributions, and (105) uses the fact that the divergence between the and density functions is .
Recalling that f_{1}(x)=\frac{1}{2}\big{(}x-\frac{1}{2}+\sqrt{2\epsilon^{\prime}}\big{)}^{2} and f_{0}(x)=\frac{1}{2}\big{(}x-\frac{1}{2}\big{)}^{2}, we have
[TABLE]
where the first inequality uses the fact that , and the second inequality follows since and hence (note that \big{(}\frac{1}{\sqrt{8}}+\frac{1}{\sqrt{2}}\big{)}^{2}\leq 2). Moreover, taking the derivatives of and gives , and substitution into (105) yields .
The preceding analysis applies in a near-identical manner when is used in place of , and yields the same KL divergence bound when is defined with respect to . As a result, for any , we obtain from (25) in Lemma 4 that . Averaging over , we obtain , and substitution into the above-established bound yields . Hence, (103) yields
[TABLE]
Now observe that if then the argument to is at least . It is easy to verify that H_{2}^{-1}\big{(}\frac{\log 2}{2}\big{)}>\frac{1}{10}, from which it follows that . Setting and noting that can be chosen arbitrarily close to , we conclude that the required number of samples recovers (101). ∎
Theorem 14 provides tight scaling laws, since stochastic gradient descent is known to achieve -optimality for strongly convex functions using O\big{(}\frac{\sigma^{2}}{\delta}\big{)} queries. Analogous results for the multi-dimensional setting can be found in [20].
7 Discussion
7.1 Limitations of Fano’s Inequality
While Fano’s inequality is a highly versatile method with successes in a wide range of statistical applications (cf., Table 1), it is worth pointing out some of its main limitations. We briefly mention some alternative methods below, as well as discussing some suitable generalizations of Fano’s inequality in Section 7.2.
Non-asymptotic weakness. Even in scenarios where Fano’s inequality provides converse bounds with the correct asymptotics including constants, these bounds can be inferior to alternative methods in the non-asymptotic sense [50, 51]. Related to this issue is the distinction between the weak converse and strong converse: We have seen that Fano’s inequality typically provides necessary conditions of the form for achieving , in contrast with strong converse results of the form for any . Alternative techniques addressing these limitations are discussed in the context of communication in [50], and in the context of statistical estimation in [52, 53].
Difficulties in adaptive settings. While we have provided examples where Fano’s inequality provides tight bounds in adaptive settings, there are several applications where alternative methods have proved to be more suitable. One reason for this is that the conditional mutual information terms (cf., Lemma 3) often involve complicated conditional distributions that are difficult to analyze. We refer the reader to [54, 55, 56] for examples in which alternative techniques proved to be more suitable for adaptive settings.
Restriction to KL divergence. When applying Fano’s inequality, one invariably needs to bound a mutual information term, which is an instance of the KL divergence. While the KL divergence satisfies a number of convenient properties that can help in this process, it is sometimes the case that other divergence measures are more convenient to work with, or can be used to derive tighter results. Generalizations of Fano’s inequality have been proposed specifically for this purpose, as we discuss in the following subsection.
7.2 Generalizations of Fano’s Inequality
Several variations and generalizations of Fano’s inequality have been proposed in the literature [57, 58, 59, 60, 61, 62]. Most of these are not derived based on the most well-known proof of Theorem 1, but are instead based on an alternative proof via the data processing inequality for KL divergence: For any event , one has
[TABLE]
where is the binary KL divergence function. Observe that if is uniform and is the event that , then we have and , and Fano’s inequality (cf., Theorem 1) follows by substituting the definition of in (108) and re-arranging. This proof lends itself to interesting generalizations, including the following.
Continuum version. Consider a continuous random variable taking values on for some , and an error probability of the form P_{\mathrm{e}}(t)=\mathbb{P}\big{[}d(V,\hat{V})>t\big{]} for some real-valued function on . This is the same formula as (11), which we previously introduced for the discrete setting. Defining the “ball” centered at , (108) leads to the following for uniform on :
[TABLE]
where denotes the volume of a set. This result provides a continuous counterpart to the final part of Theorem 2, in which the cardinality ratio is replaced by a volume ratio. We refer the reader to [35] for example applications, and to [62] for the simple proof outlined above.
Beyond KL divergence. The key step (108) extends immediately to other measures that satisfy the data processing inequality. A useful class of such measures is the class of -divergences: D_{f}(P\|Q)=\mathbb{E}_{Q}\big{[}f\big{(}\frac{P(\mathbf{Y})}{Q(\mathbf{Y})}\big{)}\big{]} for some convex function satisfying . Special cases include KL divergence (), total variation (), squared Hellinger distance (), and -divergence (). It was shown in [60] that alternative choices beyond the KL divergence can provide improved bounds in some cases. Generalizations of Fano’s inequality beyond -divergences can be found in [61].
Non-uniform priors. The first form of Fano’s inequality in Theorem 1 does not require to be uniform. However, in highly non-uniform cases where , the term may be too large for the bound to be useful. In such cases, it is often useful to use different Fano-like bounds based on the alternative proof above. In particular, the step (108) makes no use of uniformity, and continues to hold even in the non-uniform case. In [57], this bound was further weakened to provide simpler lower bounds for non-uniform settings with discrete alphabets. Fano-type lower bounds in continuous Bayesian settings with non-uniform priors arose more recently, and are typically more technically challenging; the interested reader is referred to [18, 63].
Appendix A Appendix
Here we provide the omitted proofs from the main body. Throughout the proofs, the random variables and are assumed to be discrete, whereas the other random variables involved, including the inputs and samples , may be continuous. In such cases, entropy quantities such as should be interpreted as being the differential entropy [34, Ch. 8], and probability functions such as should be interpreted as being a probability density function (PDF).
A.1 Preliminary Information-Theoretic Results
The following lemma states some useful results from information theory. The proofs can be found in standard references such as [34].
Lemma 8**.**
(Standard information-theoretic results)* We have the following:*
- •
(Chain rule for entropy)* .*
- •
(Chain rule for mutual information)* .*
- •
(Sub-additivity of entropy)* .*
- •
(Conditioning reduces entropy)* . *
- •
(Information-preserving transform)* If depends on only through , then , and .*
- •
(Capacity of binary symmetric channel)* If are binary with for (where denotes modulo-2 addition), then .*
- •
(Divergence between independent pairs)* .*
- •
(Divergence between equal-variance univariate Gaussians)* For and , it holds that . *
We will make use of these results without necessarily referencing the lemma.
A.2 Proof of Theorem 1 (Fano’s Inequality)
Defining the error indicator random variable , we have
[TABLE]
where (110) holds since is a deterministic function of , (111) follows from the chain rule, (112) holds since conditioning reduces entropy, (113) uses , and (114) follows since has no uncertainty given when , and takes one of values given when .
In case that is uniform, we obtain (7) by upper bounding and in (6), subtracting on both sides, and taking the negative on both sides.
A.3 Proof of Theorem 2 (Fano’s Inequality with Approximate Recovery)
Define the error event . Following the steps (110)–(113) with in place of , we obtain
[TABLE]
where (116) follows since when is given and , takes one of at most values, whereas if is given and , takes one of at most values. We have thus proved (14).
In case that is uniform, we obtain (15) by upper bounding and in (14), subtracting on both sides, and taking the negative on both sides.
A.4 Proof of Lemma 1 (Data Processing Inequality)
We focus on the first part, since the second and third parts follow as special cases. We have
[TABLE]
where (119) follows since conditioning reduces entropy, and (120) holds because and are conditionally independent given .
A.5 Proof of Lemma 2 (Tensorization)
We start with the second claim, since the first claim then follows by letting each deterministically equal an arbitrary fixed value (e.g., zero). To prove the second claim, we write
[TABLE]
where (123) follows from the sub-additivity of entropy and the fact that conditioning reduces entropy, (124) follows from the conditional independence of the given , and (125) follows from the assumption that depends on only on through .
The third claim follows from the second claim by writing
[TABLE]
by the assumption that depends on only through .
A.6 Proof of Lemma 3 (Tensorization with Adaptivity)
We have the following:
[TABLE]
where (128) follows from the chain rule, (129) follows since is a function of , (131) follows since is conditionally independent of given , and (132) follows since conditioning reduces entropy. This completes the proof of the first part.
To prove the second part, we note that
[TABLE]
by the assumption that depends on only through .
A.7 Proof of Lemma 5 (Covering-Based Mutual Information Bound)
Applying (25) in Lemma 4 with the choice , and letting denote expectation with respect to , we have
[TABLE]
where (136) applies the definition of KL divergence, (137) lower bounds the summation by the single term achieving the minimum in (30), and (139) applies the upper bound in (30).
A.8 Omitted Details in Discrete Examples with Approximate Recovery
A.8.1 Group Testing
Here we characterize the asymptotic behavior of the logarithm in (40). The main step is to upper bound the summation in the denominator, which is given by . By the assumption , the value must yield the highest value of when is sufficiently large. Hence, upper bounding the summation by times the maximum yields . Applying , we deduce that
[TABLE]
Since and does not depend on , a simple asymptotic analysis yields \log\sum_{j=0}^{\lfloor\alpha k\rfloor}{p-L\choose j}{L\choose k-j}\leq\big{(}\alpha k\log\frac{p}{k}+(1-\alpha)k\log\frac{L}{k}\big{)}(1+o(1)). The logarithm in (40) therefore simplifies to \big{(}k\log\frac{p}{L}\big{)}(1+o(1)), as desired.
A.8.2 Graphical Model Selection
Here we upper bound the quantity in (54). For all , the first combinatorial term is upper bounded by , the second is maximized by (for sufficiently large ), and further upper bounding yields . Taking the logarithm and applying along with asymptotic simplifications, we find that N_{\mathrm{\max}}(\alpha p)\leq\big{(}\alpha p\log p\big{)}(1+o(1)), as desired.
A.9 Proof of Theorem 10 (Reduction to Approximate Recovery)
We adopt the same general approach as Theorem 9, but instead of the error probability , we consider an approximate recovery version of the form . We again start with (66), which we repeat here:
[TABLE]
for any . Consider the following minimum-distance rule for :
[TABLE]
and suppose that we have for the correct index . Then for any such that , we have
[TABLE]
where (143) follows from the triangle inequality, and (144) follows from (72) and the assumption . As a result, when , the minimum-distance rule (142) must output some satisfying , yielding
[TABLE]
With the above tools in place, we proceed as follows:
[TABLE]
where (146) follows by maximizing over a smaller set, (147) follows from (145), (148) lower bounds the maximum by the average, and (149) follows from Fano’s inequality for approximate recovery (cf., Theorem 2) and the fact that by the data processing inequality (cf., Lemma 1). The proof of (73) is concluded by substituting (149) into (141) with , and taking the infimum over all estimators .
A.10 Proof of Theorem 11 (Reduction for Noisy Optimization)
We follow a similar proof to that of Theorem 9, which gave an analogous result for estimation. First, by Markov’s inequality, we have
[TABLE]
We proceed by analyzing the probability on the right-hand side.
Suppose that a random index is drawn uniformly from , and the triplet is generated by running the optimization algorithm on . Moreover, given , let index the function among with the lowest corresponding value: . By the assumption that any point satisfies at most one of the functions, we find that if , then it must hold that . Hence, we have
[TABLE]
where is a shorthand for .
With the above tools in place, we proceed as follows:
[TABLE]
where (152) follows by maximizing over a smaller set, (153) follows from (151), (154) lower bounds the maximum by the average, and (155) follows from Fano’s inequality (cf., (8) in Theorem 1) and the fact that by the data processing inequality (cf., third part of Lemma 3). The proof of (79) is concluded by substituting (155) into (150) and taking the infimum over all . For , we obtain (64) in the same way upon replacing (155) by the version of Fano’s inequality for given in Remark 1.
Acknowledgments
J. Scarlett was supported by an NUS startup grant. V. Cevher was supported by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement 725594 – time-data).
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1[1] R. M. Fano, “Class notes for MIT course 6.574: Transmission of information,” 1952.
- 2[2] M. Malyutov, “The separating property of random matrices,” Math. Notes Acad. Sci. USSR , vol. 23, no. 1, pp. 84–91, 1978.
- 3[3] G. Atia and V. Saligrama, “Boolean compressed sensing and noisy group testing,” IEEE Trans. Inf. Theory , vol. 58, no. 3, pp. 1880–1901, March 2012.
- 4[4] M. Wainwright, “Information-theoretic limits on sparsity recovery in the high-dimensional and noisy setting,” IEEE Trans. Inf. Theory , vol. 55, no. 12, pp. 5728–5741, Dec. 2009.
- 5[5] E. J. Candes and M. A. Davenport, “How well can we estimate a sparse vector?” Appl. Comp. Harm. Analysis , vol. 34, no. 2, pp. 317–323, 2013.
- 6[6] H. Hassanieh, P. Indyk, D. Katabi, and E. Price, “Nearly optimal sparse Fourier transform,” in ACM Symp. Theory Comp. (STOC) , 2012, pp. 563–578.
- 7[7] V. Cevher, M. Kapralov, J. Scarlett, and A. Zandieh, “An adaptive sublinear-time block sparse Fourier transform,” in ACM Symp. Theory Comp. (STOC) , 2017.
- 8[8] A. A. Amini and M. J. Wainwright, “High-dimensional analysis of semidefinite relaxations for sparse principal components,” Annals Stats. , vol. 37, no. 5B, pp. 2877–2921, 2009.