Samplers and Extractors for Unbounded Functions
Rohit Agrawal
TL;DR
This paper introduces explicit constructions of subgaussian and subexponential samplers for unbounded functions, extending extractor notions to KL-divergence, matching the best parameters of bounded function samplers.
Contribution
It develops a new framework using KL-divergence based extractors, providing the first explicit constructions of subgaussian samplers for unbounded functions.
Findings
Constructed explicit subgaussian samplers matching bounded case parameters.
Extended extractor notions to KL-divergence, strengthening the theoretical foundation.
Showed existence of KL-extractors with parameters comparable to standard extractors.
Abstract
Blasiok (SODA'18) recently introduced the notion of a subgaussian sampler, defined as an averaging sampler for approximating the mean of functions such that has subgaussian tails, and asked for explicit constructions. In this work, we give the first explicit constructions of subgaussian samplers (and in fact averaging samplers for the broader class of subexponential functions) that match the best-known constructions of averaging samplers for -bounded functions in the regime of parameters where the approximation error and failure probability are subconstant. Our constructions are established via an extension of the standard notion of randomness extractor (Nisan and Zuckerman, JCSS'96) where the error is measured by an arbitrary divergence rather than total variation distance, and a generalization of Zuckerman's…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Samplers and Extractors for Unbounded Functions
Rohit Agrawal John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138, USA. Email: [email protected]. Supported by the Department of Defense (DoD) through the National Defense Science & Engineering Graduate Fellowship (NDSEG) Program.
Abstract
Błasiok (SODA’18) recently introduced the notion of a subgaussian sampler, defined as an averaging sampler for approximating the mean of functions such that has subgaussian tails, and asked for explicit constructions. In this work, we give the first explicit constructions of subgaussian samplers (and in fact averaging samplers for the broader class of subexponential functions) that match the best known constructions of averaging samplers for -bounded functions in the regime of parameters where the approximation error and failure probability are subconstant. Our constructions are established via an extension of the standard notion of randomness extractor (Nisan and Zuckerman, JCSS’96) where the error is measured by an arbitrary divergence rather than total variation distance, and a generalization of Zuckerman’s equivalence (Random Struct. Alg.’97) between extractors and samplers. We believe that the framework we develop, and specifically the notion of an extractor for the Kullback–Leibler (KL) divergence, are of independent interest. In particular, KL-extractors are stronger than both standard extractors and subgaussian samplers, but we show that they exist with essentially the same parameters (constructively and non-constructively) as standard extractors.
1 Introduction
1.1 Averaging samplers
Averaging (or oblivious) samplers, introduced by Bellare and Rompel [BR94], are one of the main objects of study in pseudorandomness. Used to approximate the mean of a -valued function with minimal randomness and queries, an averaging sampler takes a short random seed and produces a small set of correlated points such that any given -valued function will (with high probability) take approximately the same mean on these points as on the entire space. Formally,
Definition 1.1** ([BR94]).**
A function is a averaging sampler if for all , it holds that
[TABLE]
where is the uniform distribution on . The number is the randomness complexity of the sampler, and is the sample complexity. A sampler is explicit if can be computed in time .
Traditionally, averaging samplers have been used in the context of randomness-efficient error reduction for algorithms and protocols, where the function is the indicator of a set (-valued), or more generally the acceptance probability of an algorithm or protocol (-valued). There has been significant effort in the literature to establish optimal explicit and non-explicit constructions of samplers, which we summarize in Table 1. We recommend the survey of Goldreich [Gol11b] for more details, especially regarding non-averaging samplers111A non-averaging sampler is an algorithm which makes oracle queries to and outputs an estimate of its average which is good with high probability, but need not simply output the average of ’s values on the queried points..
However, averaging samplers can also have uses beyond bounded functions: Błasiok [Bła18b], motivated by an application in streaming algorithms, introduced the notion of a subgaussian sampler, which he defined as an averaging sampler for functions such that is a subgaussian random variable. Since subgaussian random variables have strong tail bounds, subgaussian functions from have a range contained in an interval of length , and thus one can construct a subgaussian sampler from a -sampler by simply scaling the error by a factor of . Unfortunately, looking at Table 1 one sees that this induces a multiplicative dependence on in the sample complexity, and for the expander walk sampler induces a dependence of in the randomness complexity. This loss can be avoided for some samplers, such as the sampler of Chor and Goldreich [CG89] based on pairwise independence (as its analysis requires only bounded variance) and (as we will show) the Ramanujan Expander Neighbor sampler of [KPS85, GW97], but Błasiok showed [Bła18a] that the expander-walk sampler does not in general act as a subgaussian sampler without reducing the error to . We remark briefly that the median-of-averages sampler of Bellare, Goldreich, and Goldwasser [BGG93] still works and is optimal up to constant factors in the subgaussian setting (since the underlying pairwise independent sampler works), but it is not an averaging samplerFootnote 1, and matching its parameters with an averaging sampler remains open in general even for -valued functions.
One of the contributions of this work is to give explicit averaging samplers for subgaussian functions (in fact even for subexponential functions that satisfy weaker tail bounds) matching the extractor-based samplers for -valued functions in Table 1 (up to the hidden polynomial in the sample complexity). This achieves the best parameters currently known in the regime of parameters where and are both subconstant, and in particular has no dependence on in the sample complexity. We also show non-constructively that subexponentially samplers exist with essentially the same parameters as -valued samplers.
Theorem 1.2** **(Informal version of
Theorems 6.1 and 6.7).
For every integer , , and , there is a function that is:
- •
an explicit subgaussian (in fact subexponential) sampler with randomness complexity and sample complexity (see Theorem 6.1)
- •
a non-constructive subexponential sampler with randomness complexity and sample complexity (see Corollary 6.7).
1.2 Randomness extractors
To prove Theorem 1.2, we develop a corresponding theory of generalized randomness extractors which we believe is of independent interest. For bounded functions, Zuckerman [Zuc97] showed that averaging samplers are essentially equivalent to randomness extractors, and in fact several of the best-known constructions of such samplers arose as extractor constructions. Formally, a randomness extractor is defined as follows:
Definition 1.3** (Nisan and Zuckerman [NZ96]).**
A function is said to be a extractor if for every distribution over satisfying , the distributions and are -close in total variation distance. Equivalently, for all it holds that . The number is called the seed length, and the output length.
The formulation of 1.3 in terms of -valued functions implies that extractors produce an output distribution that is indistinguishable from uniform by all bounded functions . It is therefore natural to consider a variant of this definition for a different set of test functions which need not be bounded.
Definition 1.4** **(Special case of 3.1 using
2.5).
A function is said to be a extractor for a set of real-valued functions from if for every distribution over satisfying and every , it holds that .
We show that much of the theory of extractors and samplers carries over to this more general setting. In particular, we generalize the connection of Zuckerman [Zuc97] to show that extractors for a class of functions of are also samplers for that class, along with the converse (though as for total variation distance, there is some loss of parameters in this direction). Thus, to construct a subgaussian sampler it suffices (and is preferable) to construct a corresponding extractor for subgaussian test functions, which is how we prove Theorem 1.2.
Unfortunately, the distance induced by subgaussian test functions is not particularly pleasant to work with: for example the point masses on [math] and in are apart, but embedding them in the larger universe leads to distributions which are apart. We solve this problem by constructing extractors for a stronger notion, the Kullback–Leibler (KL) divergence, equivalently, extractors whose output is required to have very high Shannon entropy.
Definition 1.5** **(Special case of 3.1 using
KL divergence).
A function is said to be a KL-extractor if for every distribution over satisfying it holds that , or equivalently .
A strong form of Pinsker’s inequality (e.g. [BLM13, Lemma 4.18]) implies that a KL-extractor is also a extractor for subgaussian test functions. The KL divergence has the advantage that is nonincreasing under the application of functions (the famous data-processing inequality), and although it does not satisfy a traditional triangle inequality, it does satisfy a similar inequality when one of the segments satisfies stronger bounds. These properties allow us to show that the zig-zag product for extractors of Reingold, Wigderson, and Vadhan [RVW00] also works for KL-extractors, and therefore to construct KL-extractors with seed length depending on and only through the entropy deficiency of rather than itself, which in the sampler perspective corresponds to a sampler with sample complexity depending on the failure probability rather than the universe size . Hence, we prove Theorem 1.2 by constructing corresponding KL-extractors.
Theorem 1.6** (Informal version of Theorem 6.2).**
For all integers , , and there is an explicit KL-extractor with , , and .
Though the above theorem is most interesting in the high min-entropy regime where , we also show the existence of KL-extractors matching most of the existing constructions of total variation extractors. In particular, we note that extractors for are immediately KL-extractors without loss of parameters, and also that any extractor can be made a KL-extractor by taking slightly smaller error, so that the extractors of Guruswami, Umans, and Vadhan [GUV09] can be taken to be KL-extractors with essentially the same parameters.
Furthermore, in addition to our explicit constructions, we also show non-constructively that KL-extractors (and hence subgaussian extractors) exist with very good parameters:
Theorem 1.7** (Informal version of Theorem 5.30).**
For any integers and there is a KL-extractor with and .
One key thing to note about the nonconstructive KL extractors of the above theorem is that they incur an entropy loss of only , whereas total variation extractors necessarily incur entropy loss by the lower bound of Radhakrishnan and Ta-Shma [RT00]. In particular, by Pinsker’s inequality, KL-extractors with the above parameters are also optimal standard (total variation) extractors [RT00], so that one does not lose anything by constructing a KL-extractor rather than a total variation extractor. We also remark that the above theorem gives subgaussian samplers with better parameters than a naive argument that a random function should directly be a subgaussian sampler, as it avoids the need to take a union bound over test functions (for ) which results in additional additive factors in the randomness complexity.
In the total variation setting, there are only a couple of methods known to explicitly achieve optimal entropy loss , the easiest of which is to use an extractor which natively has this sort of loss, of which only three are known: An extractor from random walks over Ramanujan Graphs due to Goldreich and Wigderson [GW97], the Leftover Hash Lemma due to Impagliazzo, Levin, and Luby [ILL89] (see also [McI87, BBR88]), and the extractor based on almost-universal hashing of Srinivasan and Zuckerman [SZ99]. Unfortunately, all of these are extractors and so must have seed length linear in (cf. [Vad12, Problem 6.4]), rather than logarithmic in as known non-constructively. The other alternative is to use the generic reduction of Raz, Reingold, and Vadhan [RRV02] which turns any extractor with entropy loss into one with entropy loss by paying an additive in seed length. We show that all of these extractors and the [RRV02] transformation also work to give KL-extractors with entropy loss , so that applications which require minimal entropy loss can also use explicit constructions of KL-extractors.
1.3 Future directions
Broadly speaking, we hope that the perspective of KL-extractors will bring new tools (perhaps from information theory) to the construction of extractors and samplers. For example, since KL-extractors can have seed length with dependence on of only , trying to explicitly construct a KL-extractor with seed length may also shed light on how to achieve optimal dependence on in the total variation setting.
In the regime of constant , we do not have explicit constructions of subgaussian samplers matching the expander-walk sampler of Gillman [Gil98] for -valued functions, which achieves randomness complexity and sample complexity , as asked for by Błasiok [Bła18b]. From the extractor point-of-view, it would suffice (by the reduction of [GW97, RVW00] that we analyze for KL-extractors) to construct explicit linear degree KL-extractors with parameters matching the linear degree extractor of Zuckerman [Zuc07], i.e. with seed length and for . A potentially easier problem, since the Zuckerman linear degree extractor is itself based on the expander-walk sampler, could be to instead match the parameters of the near-linear degree extractors of Ta-Shma, Zuckerman, and Safra [TZS06] based on Reed–Muller codes, thereby achieving sample complexity .
Finally, we hope that KL-extractors can also find uses beyond being subgaussian samplers and total variation extractors: for example it seems likely that there are applications (perhaps in coding or cryptography, cf. [BDK*+*11]) where it is more important to have high Shannon entropy in the output than small total variation distance to uniform, in which case one may be able to use KL-extractors with entropy loss only directly, rather than a total variation extractor or KL-extractor with entropy loss .
2 Preliminaries
2.1 (Weak) statistical divergences and metrics
Our results in general will require very few assumptions on notions of “distance” between probability distributions, so we will give a general definition and indicate in our theorems when we need which assumptions.
Definition 2.1**.**
A weak statistical divergence (or simply weak divergence) on a finite set is a function from pairs of probability distributions over to . We write for the value of on distributions and . Furthermore
If with equality iff , then is positive-definite, and we simply call a divergence. 2. 2.
If , then is symmetric. 3. 3.
If , then satisfies the triangle inequality. 4. 4.
If for all , then is jointly convex. If this holds only when then is convex in its first argument. 5. 5.
If is defined on all finite sets and for all functions the divergence is nonincreasing under , that is , then satisfies the data-processing inequality.
If is positive-definite, symmetric, and satisfies the triangle inequality, then it is called a metric.
Example 2.2**.**
The distance for between probability distributions over is
[TABLE]
and is positive-definite and symmetric. Furthermore, for it satisfies the triangle inequality (and so is a metric), and is jointly convex. The distance is nonincreasing in .
Example 2.3**.**
The total variation distance is
[TABLE]
and is a jointly convex metric that satisfies the data-processing inequality.
Example 2.4** (Rényi Divergences [Rén61]).**
For two probability distributions and over a finite set , the Rényi -divergence or Rényi divergence of order is defined for real by
[TABLE]
where the logarithm is in base (as are all logarithms in this paper unless noted otherwise). The Rényi divergence is continuous in and so is defined by taking limits for , giving for the divergence , for the Kullback–Leibler (or KL) divergence
[TABLE]
and for the max-divergence . The Rényi divergence is nondecreasing in . Furthermore, when the Rényi divergence is jointly convex, and for all the Rényi divergence satisfies the data-processing inequality [vEH14].
When is the uniform distribution over the set , then for all , where is called the Rényi -entropy of . For , is the max-entropy of , for , is the Shannon entropy of , and for , is the min-entropy of .
For , the Rényi -entropy can be expressed in terms of the -distance to uniform:
[TABLE]
2.2 Statistical weak divergences from test functions
Zuckerman’s connection [Zuc97] between samplers for bounded functions and extractors for total variation distance is based on the following standard characterization of total variation distance as the maximum distinguishing advantage achieved by bounded functions,
[TABLE]
By considering an arbitrary class of functions in the supremum, we get the following weak divergence:
Definition 2.5**.**
Given a finite and a set of real-valued functions , the -distance on between probability measures on is denoted by and is defined as
[TABLE]
where we use a superscript to avoid confusion with the Csiszár-Morimoto-Ali-Silvey -divergences [Csi63, Mor63, AS66].
We call the set of functions symmetric if for all there is and such that , and distinguishing if for all there exists with .
Example 2.6**.**
If or , then is exactly the total variation distance.
Remark 2.7*.*
An equivalent definition of being symmetric is that for all there exists with for all distributions and . Hence, one might also consider a weaker notion of symmetry that reverses quantifiers, where is “weakly-symmetric” if for all and distributions and there exists such that . However, such a class gives exactly the same weak divergence as its “symmetrization” , so we do not need to introduce this more complex notion.
Remark 2.8*.*
By identifying distributions with their probability mass function, one can realize as an inner product . 2.5 can thus be written as , which is essentially the notion of indistinguishability considered in several prior works, (see e.g. the survey of Reingold, Trevisan, Tulsiani, and Vadhan [RTTV08]), but without requiring all to be bounded.
Remark 2.9*.*
For simplicity, all our probabilistic distributions are given only for random variables and distributions over finite sets as this is all we need for our application. A more general version of 2.5 has been studied by e.g. Zolotarev [Zol84] and Müller [Mül97] and is commonly used in developments of Stein’s method in probability.
We now establish some basic properties of .
Lemma 2.10**.**
Let be a set of real-valued functions over a finite set . Then satisfies the triangle inequality and is jointly convex, and
if is symmetric then is symmetric and
[TABLE] 2. 2.
if is distinguishing then is positive-definite,
so that if is both symmetric and distinguishing then is a jointly convex metric on probability distributions over , in which case we also use the notation .
Proof.
The triangle inequality and joint convexity both follow from the linearity of each , as by linearity of expectation, for all it holds that
[TABLE]
Upper bounding the terms on the right-hand side by and taking the supremum of the left hand side over then gives the claims. The symmetry and positive-definite claims are immediate from the definitions. ∎
Furthermore, the notion of dual norm has an appealing interpretation in this framework via 2.8, generalizing the fact that total variation distance corresponds to -valued test functions (or equivalently that distance corresponds to to -valued functions).
Proposition 2.11**.**
Let be Hölder conjugates (meaning ), and let
[TABLE]
be the set of real-valued functions from with bounded -th moments. Then , in the sense that for all probability distributions and over it holds that .
In particular, taking and recovers the result for (equivalently total variation) distance.
Proof.
As mentioned this is just the standard fact that the and norms are dual, but for completeness we include a proof in our language using the extremal form of Hölder’s inequality (note that since we are dealing with finite probability spaces the extremal equality holds even for and ). Given probability distributions and over , we have that
[TABLE]
as desired. ∎
3 Extractors for weak divergences and connections to samplers
3.1 Definitions
We now use this machinery to extend the notion of an extractor due to Nisan and Zuckerman [NZ96] and the average-case variant of Dodis, Ostrovsky, Reyzin, and Smith [DORS08].
Definition 3.1** (Extends 1.4).**
Let be a weak divergence on the set , and . Then if for all distributions over with it holds that
, then is said to be a extractor for , or a -extractor. 2. 2.
, then is said to be a strong extractor for , or a strong -extractor.
Furthermore, if for all joint distributions where is distributed over with , it holds that
, then is said to be a average-case extractor for , or a average-case -extractor. 2. 4.
, then is said to be a average-case strong extractor for , or a average-case strong -extractor.
Remark 3.2*.*
By taking to be the total variation distance we recover the standard definitions of extractor and strong extractor due to [NZ96] and the definition of average-case extractor due to [DORS08].
However, our definitions are phrased slightly differently for strong and average-case extractors as an expectation rather than a joint distance, that is, for strong average-case extractors we require a bound on the expectation rather than a bound on . In our setting, the weak divergence need not be defined over the larger joint universe, but it is defined for all random variables over . In the case of and KL divergence, both definitions are equivalent (for KL divergence, this is an instance of the chain rule).
Remark 3.3*.*
The strong variants of 3.1 are also non-strong extractors assuming the weak divergence is convex in its first argument, as it is for most weak divergences of interest, including the norms for , all defined by test functions, the KL divergence, Rényi divergences for , and all Csiszár-Morimoto-Ali-Silvey -divergences. The average-case variants are always non-average-case extractors by taking to be independent of .
Remark 3.4*.*
We gave 3.1 for general weak divergences which need not be symmetric, and made the particular choice that the output of the extractor was on the left-hand side of the weak divergence and that the uniform distribution was on the right-hand side. This is motivated by the standard information-theoretic divergences such as KL divergence, which require the left-hand distribution to have support contained in the support of the right-hand distribution, and putting the uniform distribution on the right ensures this is always the case. Furthermore, the KL divergence to uniform has a natural interpretation as an entropy difference, for the Shannon entropy, so that in particular a KL extractor with error requires the output to have Shannon entropy at least . If for a weak divergence the other direction is more natural, one can always reverse the sides by considering the weak divergence .
Remark 3.5*.*
3.1 does not technically need even a weak divergence, as it suffices to simply have a measure of distance to uniform. However, since weak divergences have minimal constraints, one can define a weak divergence from any distance to uniform by ignoring the second component (or setting it to be infinite for non-uniform distributions).
We also give the natural definition of averaging samplers for arbitrary classes of functions extending 1.1, along with the strong variant of Zuckerman [Zuc97].
Definition 3.6**.**
Given a class of functions , a function is said to be a strong averaging sampler for or a strong averaging -sampler if for all , it holds that
[TABLE]
where . If this holds only when , then it is called a (non-strong) averaging sampler for or averaging -sampler. We say that is a strong absolute averaging sampler for if it also holds that
[TABLE]
with the analogous definition for non-strong samplers.
Remark 3.7*.*
We separated a single-sided version of the error bound in 3.6 as in [Vad12], as it makes the connection between extractors and samplers cleaner and allows us to be specific about what assumptions are needed. Note that if is symmetric then every (strong) sampler for is a (strong) absolute sampler for , recovering the standard notion up to a factor of in .
3.2 Equivalence of extractors and samplers
We now show that Zuckerman’s connection [Zuc97] does indeed generalize to this broader setting as promised.
Theorem 3.8**.**
Let be an -extractor (respectively strong extractor) for the weak divergence defined by a class of test functions as in 2.5. Then the function for defined by is a -sampler (respectively strong sampler) for .
Proof.
The proof is essentially the same as that of [Zuc97].
Fix a collection of test functions , where if is not strong we restrict to , and let be defined as
[TABLE]
where is the point mass on . Then if is uniform over , we have
[TABLE]
Since is an -extractor (respectively strong extractor) for we must have . But by definition, so we have . Hence, the probability that a random lands in is less than , and since is exactly the set of seeds which are bad for , this concludes the proof. ∎
Remark 3.9*.*
Hölder’s inequality implies that an extractor for with error is also an extractor and thus -averaging sampler with error . 2.11 and 3.8 show that they are in fact samplers for the much larger class of functions with bounded moments (rather than just moments), also with error .
Furthermore, if all the functions in have bounded deviation from their mean (for example, subgaussian functions from have such a bound of by the tail bounds from Lemma 4.3), then we also have a partial converse that recovers the standard converse in the case of total variation distance.
Theorem 3.10**.**
Let be a class of functions with finite maximum deviation from the mean, meaning \operatorname{max\,dev}(\mathcal{F})\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny def}}}{=}}\sup_{f\in\mathcal{F}}\max_{x\in\left\{0,1\right\}^{n}}\MT@delim@Swapped\of@nostar[\big{]}{f(x)-\operatorname*{\mathbb{E}}\MT@delim@Swapped\brof@star{f(U_{m})}}<\infty. Then given a -sampler (respectively strong -sampler) , the function for defined by is a \MT@delim@Swapped\of@nostar[\big{]}{k,\varepsilon+\delta\cdot 2^{n-k}\cdot\operatorname{max\,dev}(\mathcal{F})} -extractor (respectively strong -extractor) for every .
In particular, is an \MT@delim@Swapped\of@nostar[\big{]}{n-\log(1/\delta)+\log(1/\eta),\varepsilon+\eta\cdot\operatorname{max\,dev}(\mathcal{F})} average-case -extractor (respectively strong average-case -extractor) for every .
Proof.
Again the proof is analogous to the one in [Zuc97].
Fix a distribution over with and a collection of test functions , where if is not strong we restrict to .
Then since is a -sampler, we know that the set of seeds for which the sampler is bad must be small. Formally, the set
[TABLE]
has size . Thus, since has min-entropy at least we know , so we have
[TABLE]
completing the proof of the main claim. The “in particular” statement follows since if are jointly distributed with we have
[TABLE]
by definition of conditional min-entropy. ∎
3.3 All extractors are average-case
Under a similar boundedness condition for general weak divergences, we can recover the standard fact that all extractors are average-case extractors under a slight loss of parameters (the same loss as achieved by Dodis, Ostrovsky, Reyzin, and Smith [DORS08] for the case of total variation distance). More interestingly, if the weak divergence is given by for a symmetric class of (possibly unbounded) functions , we can also generalize and recover the result of Vadhan [Vad12, Problem 6.8] that shows that a extractor (for total variation) is a average-case extractor without any other loss.
Theorem 3.11**.**
Let be a bounded weak divergence over , meaning that
[TABLE]
Then a -extractor for (respectively strong extractor) is also a average-case-extractor for (respectively strong average-case-extractor) for any .
Proof.
The proof is analogous to that of [DORS08]. We prove it only for non-strong extractors, the proof for strong extractors is completely analogous by adding more expectations.
For jointly distributed random variables such that , we have by [DORS08, Lemma 2.2] that the probability that . Thus
[TABLE]
Theorem 3.12**.**
Let be a symmetric class of test functions and be a extractor (respectively strong extractor) for , where is at most . Then is an average-case extractor (respectively strong average-case extractor) for .
Remark 3.13*.*
Theorem 3.12 also applies to extractors for the norms via 2.11.
The proof of Theorem 3.12 follows the strategy outlined by Vadhan [Vad12, Problem 6.8]. We first isolate the following key lemma which shows that any extractor with error that gracefully decays with lower min-entropy is average-case with minimal loss of parameters, as opposed to Theorem 3.11 which used a worst-case error bound when the min-entropy is low.
Lemma 3.14**.**
Let be a extractor (respectively strong extractor) for such that for every , is also a extractor (respectively strong extractor) for . Then is a average-case extractor (respectively strong average-case extractor) for .
Proof.
We prove this for strong extractors, the non-strong case is analogous. For every with distributed on and , we have
[TABLE]
where the last inequality follows from the fact that by definition of conditional min-entropy. ∎
Proof of Theorem 3.12.
By the previous lemma, it suffices to prove that for every , is a extractor (respectively strong extractor) for . Since is convex in its first argument by Lemma 2.10, following Chor and Goldreich [CG88] it is enough to consider only distributions with min-entropy that are supported on a set of at most . Fix such a distribution and a collection of test functions with if is not strong. Then since is supported on a set of size at most , the distribution that is uniform over the complement of has min-entropy at least , and furthermore the mixture has min-entropy at least . Hence, as is a extractor (respectively strong extractor) for ,
[TABLE]
where is such that as guaranteed to exist by the symmetry of . ∎
4 Subgaussian distance and connections to other notions
Now that we’ve introduced the general machinery we need, we can go back to our motivation of subgaussian samplers. We will need some standard facts about subgaussian and subexponential random variables, we recommend the book of Vershynin [Ver18] for an introduction.
Definition 4.1**.**
A real-valued mean-zero random variable is said to be subgaussian with parameter if for every the moment generating function of is bounded as
[TABLE]
If this is only holds for then is said to be -subgamma, and if is -subgamma then is said to be subexponential with parameter .
Remark 4.2*.*
There are many definitions of subgaussian (and especially subexponential) random variables in the literature, but they are all equivalent up to constant factors in and only affect constants already hidden in big-’s.
Lemma 4.3**.**
Let be a real-valued random variable. Then
(Hoeffding’s lemma) If is bounded in the interval , then is subgaussian with parameter . 2. 2.
If is mean-zero, then is subgaussian (respectively subexponential) with parameter if and only if is subgaussian (respectively subexponential) with parameter for every .
Furthermore, if is mean-zero and subgaussian with parameter , then
For all , \max\MT@delim@Swapped\of@nostar[\big{]}{\Pr\MT@delim@Swapped\brof@star{Z>t},\Pr\MT@delim@Swapped\brof@star{Z<-t}}\leq e^{-t^{2}/2\sigma^{2}}. 2. 2.
* for all .* 3. 3.
* is subexponential with parameter .*
We are now in a position to formally define the subgaussian distance.
Definition 4.4**.**
For every finite set , we define the set of subgaussian test functions on (respectively the set of subexponential test functions on ) to be the set of functions such that the random variable is mean-zero and subgaussian (respectively subexponential) with parameter . Then and are symmetric and distinguishing, so by Lemma 2.10 the respective distances induced by and are jointly convex metrics called the subgaussian distance and subexponential distance respectively and are denoted as and .
Remark 4.5*.*
We choose subgaussian parameter in 4.4 as by Hoeffding’s lemma, all functions have that is subgaussian with parameter , so this choice preserves the same “scale” as total variation distance. However, the choice of parameter is essentially irrelevant by linearity, as different choices of parameter simply scale the metric .
Note that absolute averaging samplers for from 3.6 are exactly subgaussian samplers as defined in the introduction. Thus, by 3.7 and 3.8, to construct subgaussian samplers it is enough to construct extractors for the subgaussian distance .
4.1 Composition
Unfortunately, the subgaussian distance has a major disadvantage compared to total variation distance that complicates extractor construction: it does not satisfy the data-processing inequality, that is, there are probability distributions and over a set and a function such that
[TABLE]
This happens because subgaussian distance is defined by functions which are required to be subgaussian only with respect to the uniform distribution. A simple explicit counterexample comes from taking defined by and taking to be the point mass on [math] and the point mass on . Their subgaussian distance in is obviously , but the subgaussian distance of and in is .
The reason this matters because a standard operation (cf. Nisan and Zuckerman [NZ96]; Goldreich and Wigderson [GW97]; Reingold, Vadhan, and Wigderson [RVW00]) in the construction of samplers and extractors for bounded functions is to do the following: given extractors
[TABLE]
define by
[TABLE]
The reason this works for total variation distance is exactly the data-processing inequality: if has enough min-entropy given , then will be close in total variation distance to , and by the data-processing inequality for total variation distance this closeness is not lost under the application of . The assumption that has min-entropy given means that is a so-called block-source, and is implied by having enough min-entropy as a joint distribution. From the sampler perspective, this construction uses the inner sampler to subsample the outer sampler. On the other hand, for subgaussian distance, the distribution can be -close to uniform but still have some element with excess probability mass , and this element (seed) when mapped by can retain222Given a subgaussian extractor with , adding a single extra seed to such that results in a subgaussian extractor with error at most by convexity of and the fact that . this excess mass in , which results in subgaussian distance . Similarly, from the sampler perspective, even when the outer sampler is a good subgaussian sampler for , there is no reason that a good subgaussian sampler for the seeds of will preserve the larger sampler property when .
Thus, since this composition operation is needed to construct high-min entropy extractors with the desired seed length even for total variation distance, to construct such extractors for subgaussian distance we need to bypass this barrier. The natural approach is to construct extractors for a better-behaved weak divergence that bounds the subgaussian distance.
Remark 4.6*.*
Similar reasoning shows that if is a strong subgaussian extractor, then it is not necessarily the case that the function that prepends the seed to the output is a (non-strong) subgaussian extractor (in contrast to extractors for total variation distance), though the converse does hold.
4.2 Connections to other weak divergences
Therefore, to aid in extractor construction, we show how relates to other statistical weak divergences.
Most basically, the subgaussian distance over differs from total variation distance up to a factor of .
Lemma 4.7**.**
Let and be distributions on . Then
[TABLE]
Proof.
That is immediate from Hoeffding’s lemma and the discussion in 4.5. The reverse bound holds since any subgaussian function takes values at most away from the mean by the tail bounds from part 3 of Lemma 4.3, and so any subgaussian test function has the property that is -valued and thus lower bounds the total variation distance. ∎
While this allows constructing subgaussian extractors and samplers from total variation extractors, as discussed in the introduction the fact that the upper bound depends on leads to suboptimal bounds. By starting with a stronger measure of error, we pay a much smaller penalty.
Lemma 4.8**.**
Let and be distributions on . Then for every
[TABLE]
In particular, that there is only an additional factor when moving to subgaussian distance compared to total variation, which in particular does not depend on and is constant for constant .
Proof.
By 2.11, for any function it holds that
[TABLE]
The result follows since -valued functions satisfy moment bounds for all , and functions which are subgaussian satisfy moment bounds by Lemma 4.3. ∎
One downside of starting with bounds on is that, extending a well-known linear seed length linear bound for -extractors (e.g. [Vad12, Problem 6.4]), we show in Corollary 5.29 that for every , there is a constant such any extractor with error smaller than requires seed length linear in , for the entropy deficiency and the output length. One might hope that sending to [math] would eliminate this linear lower bound but still bound the subgaussian distance, but phrased this way sending to [math] just results in a total variation extractor.
However, with a shift in perspective essentially the same approach works: by 2.4, implies , and there is an analogous linear seed length lower bound on constant error extractors for every . In this case, however, sending to [math] results in the KL divergence, which does upper bound the subgaussian distance, and in fact with the same parameters as for total variation distance.
Lemma 4.9**.**
Let be a distribution on . Then
[TABLE]
where is the (concave) binary entropy function.
Proof.
The upper bound on subgaussian distance follows from a general form of Pinsker’s inequality as in [BLM13, Lemma 4.18], but for the extension to subexponential functions we reproduce its proof here, based on the Donsker–Varadhan “variational” formulation of KL divergence [DV76] (cf. [BLM13, Corollary 4.15])
[TABLE]
Now if satisfies , then by letting , this implies
[TABLE]
for all . Thus, when , we have .
Then since subgaussian random variables satisfy such a bound for all , we can make the optimal choice to get the claimed bound on . For subexponential random variables, which satisfy such a bound only for , we choose , which gives
[TABLE]
as desired. The concavity of this bound follows by noting that it has a continuous and nonincreasing derivative.
For the reverse inequality, we use a bound on the difference in entropy between distributions and on a set of size which states
[TABLE]
This inequality is a simple consequence of Fano’s inequality as noted by Goldreich and Vadhan [GV99, Fact B.1], and implies the desired result by taking as and . ∎
Remark 4.10*.*
There are sharper upper bounds on the KL divergence than given in Lemma 4.9, such as the bound of Audenaert and Eisert [AE05, Theorem 6], but the bound we use has the advantage of being defined for the entire range of the total variation distance and being everywhere concave.
5 Extractors for KL divergence
By Lemma 4.9, the subgaussian distance can be bounded in terms of the KL divergence to uniform, so by the following easy lemma to construct subgaussian extractors it suffices to construct extractors for KL divergence.
Lemma 5.1**.**
Let and be weak divergences on the set and be a function such that for all distributions on . Then if is increasing on , every extractor for is also a -extractor for , and if is also concave, then if is strong or average-case as a -extractor, it has the same properties as a extractor for .
Importantly, the KL divergence does not have the flaws of subgaussian distance discussed in Section 4.1. The classic data-processing inequality says that KL divergence is non-increasing under postprocessing by (possibly randomized) functions, and the chain rule for KL divergence says that
[TABLE]
for all distributions , , , and , so that in particular
[TABLE]
and prepending the seed of a strong KL-extractor does in fact give a non-strong KL-extractor:
Lemma 5.2**.**
A function is a strong KL-extractor (respectively strong average-case KL-extractor) if and only if the function defined by is a (non-strong) KL-extractor (respectively average-case KL-extractor).
Furthermore, KL divergence satisfies a type of triangle inequality when combined with higher Rényi divergences:
Lemma 5.3**.**
Let , , and be distributions over a finite set . Then for all , it holds that
[TABLE]
Proof.
This follows from a characterization of Rényi divergence due to van Erven and Harremoës [vE10, Lemma 6.6] [vEH14, Theorem 30] and Shayevitz [Sha11, Theorem 1], who prove that for for every positive real and distributions and that
[TABLE]
In particular, choosing , , and and upper bounding the infimum by the particular choice of gives the claim. ∎
5.1 Composition
These properties imply that composition does work as we want (without any loss depending on the output length ) assuming we have extractors for KL and higher divergences.
Theorem 5.4** (Composition for high min-entropy Rényi entropy extractors, cf. [GW97]).**
Suppose
* is an extractor for with ,* 2. 2.
* is an average-case KL-extractor,*
and define by \operatorname{Ext}\MT@delim@Swapped\of@nostar[\big{]}{(x,y),s}=\operatorname{Ext}_{out}(x,\operatorname{Ext}_{in}(y,s)). Then is an extractor for KL. Furthermore, if is a strong average-case KL-extractor, then is a strong KL-extractor, and if is average-case then so is .
Proof.
Let be jointly distributed random variables with distributed over and over such that . Let be a distribution over which is independent of , , and . Then for every , we have by Lemma 5.3 and the data-processing inequality for KL divergence that
[TABLE]
where the last equality follows from the chain rule for KL divergence. Now by standard properties of conditional min-entropy (see for example [DORS08, Lemma 2.2]), we know that and .
If is not average-case, take to be a constant independent of and , and if is average-case then take the average of both sides over . The claim for non-strong then follows by taking which bounds the first term by and the second by . The claim for strong follows by choosing to be the point mass on and then taking the expectation of both sides over a uniform . ∎
Remark 5.5*.*
Theorem 5.4 in fact a construction of a block-source KL-extractor, meaning that the claimed error bounds hold for any joint distributions such that and rather than just those distributions with . The extra entropy loss inherent in the non-block analysis is why Reingold, Wigderson, and Vadhan [RVW00] introduced the zig-zag product for extractors, which we will apply for KL-extractors in Corollary 5.19.
5.2 Existing explicit constructions
The construction of Theorem 5.4 required both a -extractor and an average-case KL-extractor, so for the result not to be vacuous we need to show the existence of such extractors. Thankfully, 2.4 implies that extractors for are also extractors for , so we can use existing extractors from the literature, such as the Leftover Hash Lemma of Impagliazzo, Levin, and Luby [ILL89] (see also [McI87, BBR88]) and its variant using almost-universal hash functions due to Srinivasan and Zuckerman [SZ99].
Proposition 5.6** ([McI87, BBR88, ILL89, IZ89, SZ99, DORS08]).**
Let be a collection of -almost universal hash functions from the set to the set , meaning that for all it holds that . Then the function defined by is an average-case -extractor.
In particular, for every and there is an explicit strong average-case extractor for (and KL) with seed length and , given by for drawn from an appropriate almost-universal hash family.
Proof.
The claim is implicit in Rackoff’s proof of the Leftover Hash Lemma (see [IZ89]) and Srinivasan and Zuckerman’s proof of the claim for total variation [SZ99], which both analyzed the collision probability of the output, and the average-case claim was proved by Dodis, Ostrovsky, Reyzin, and Smith [DORS08], though we include a proof here for completeness.
Given a joint distribution such that is distributed over with , we have
[TABLE]
The in particular statement follows from Lemma 5.7 below and from the existence of -almost universal hash families with size as constructed by [SZ99]. ∎
To establish the claim about strong extractors, we generalize Lemma 5.2 to extractors for for :
Lemma 5.7**.**
If is a -extractor (respectively average-case -extractor) for such that , then is a strong -extractor (respectively strong average-case -extractor).
Proof.
[TABLE]
Following Vadhan [Vad12], we also note that the extractor based on expander walks due to Goldreich and Wigderson [GW97], which has the nice property that its seed length depends only on the entropy deficiency of the source rather than itself, is also an extractor. Before stating the extractor formally, we introduce some notation and terminology we will need.
Definition 5.8**.**
Let be a -regular graph on with adjacency matrix and transition matrix . Then if has eigenvalues , the spectral expansion of is . A function is a neighbor function of if there is some labelling of the edges of for which is the vertex obtained by following the th edge out of in . is consistently labelled if for all and we have , that is, at most one incoming edge is labelled by .
Lemma 5.9**.**
Let be the neighbor function of a graph with spectral expansion . Then for every , is a -extractor and a -extractor. Furthermore, if is consistently labelled, then the function is such that is an injection out of .
In particular, if then is an average-case -extractor and an average-case -extractor.
Proof.
If is a distribution over with , then so that . The -extractor result follows since the action of reduces the distance to uniform by a factor of , and the -extractor result from the fact that for every distribution on .
For the furthermore claim, we need to show that is an injection, or equivalently that given and , one can recover . But by definition of consistent labelling, at most one edge into is labelled by , and so taking this edge from gives , as desired. Finally, the in particular claim follows by Jensen’s inequality, since and square-root are concave, and by definition. ∎
Remark 5.10*.*
The fact that is an injection implies that, unlike for the extractors from hashing of 5.6, the result of prepending the seed to the output of the expander-walk extractor does not give a extractor. However, it will be very useful in concert with Reingold, Vadhan, and Wigderson’s zig-zag product for extractors [RVW00] to avoid the entropy loss in Theorem 5.4.
Corollary 5.11** **([GW97] [Vad12, Discussion after Theorem
6.22]).
There is a universal constant such that for every , , and there is an explicit average-case -extractor (respectively average-case -extractor) with such that the function is an injection.
Moreover, if there is an explicit construction of consistently labelled neighbor functions for Ramanujan graphs over with degree , then one can take .
Proof.
By Lemma 5.9 it suffices to demonstrate the existence of an explicit -regular expander graph over with a consistently labelled neighbor function , spectral expansion , and . The claim about Ramanujan graphs is thus immediate since a Ramanujan graph with degree has .
Without the assumption of good Ramanujan graphs, we can use a power of the the explicit constant degree expander of Margulis–Gabber–Galil [Mar73, GG81] (technically this requires even, which following Goldreich [Gol11a] we can fix when is odd by joining two graphs on by the canonical perfect matching, and we can add self-loops to ensure the degree is a power of ). This graph is consistently labelled with degree and constant spectral expansion . Then the graph on with edges representing -length paths has spectral expansion and degree , which for gives and degree for as desired. ∎
We argued that the above extractors are KL-extractors using the fact they are (and thus extractors, but one can also show that any total variation extractor with sufficiently small error is a KL-extractor, albeit with some loss of parameters.
Lemma 5.12**.**
For every extractor for total variation distance such that , is also a -KL-extractor, where is the binary entropy function. Furthermore, if is strong, average-case, or both as a total variation extractor, then it has the same properties as a KL-extractor.
In particular, if , then every extractor (respectively strong extractor) is an average-case KL-extractor (respectively strong average-case KL-extractor).
Proof.
The main claim is an immediate corollary of Lemmas 4.9 and 5.1. The in particular statement follows since being a extractor (respectively strong extractor) implies by Theorem 3.12 that it is a average-case (respectively strong average-case) extractor, so since we have chosen to make , we know is an average-case KL-extractor (respectively strong average-case KL-extractor). ∎
Remark 5.13*.*
Reducing by a factor of increases the seed length and entropy loss of the input extractor. For the former, this is often (but not always) tolerable since the input extractor may already depend suboptimally on . For the latter, we will show in Corollary 5.21 how to use the transform of Raz, Reingold, and Vadhan [RRV02] to recover bits of lost entropy (at least this much must be lost by Radhakrishnan and Ta-Shma [RT00]) at a cost of in the seed length.
Instantiating Lemma 5.12 with the Guruswami–Umans–Vadhan [GUV09] extractor for total variation distance, we see that the increased seed length and entropy loss are simply absorbed into the existing hidden constants:
Theorem 5.14** (KL-analogue of [GUV09, Theorem 1.5]).**
For every , , and , there is an explicit average-case (respectively strong average-case) KL-extractor with and (respectively ).
5.3 Reducing the entropy loss of KL-extractors
In this section, we show how to avoid the entropy loss inherent in Theorem 5.4 using the zig-zag product for extractors, introduced by Reingold, Vadhan, and Wigderson [RVW00]. This product combines a technique of Raz and Reingold [RR99] to preserve entropy and the method of Wigderson and Zuckerman [WZ99] to extract entropy left over in a source after an initial extraction, and we show that these techniques extend to the setting of KL-extractors. Furthermore, these techniques (along with the Leftover Hash Lemma) are also the key to the transformation of Raz, Reingold, and Vadhan [RRV02] to convert an arbitrary extractor into one with optimal entropy loss, so we show that this transformation works for KL-extractors as well.
For all of these results, the key is the following lemma:
Lemma 5.15** (Re-extraction from leftovers).**
Let
* be a KL-extractor,* 2. 2.
* be a function such that is an injective map,* 3. 3.
* be a average-case KL-extractor for .*
Then defined by \operatorname{Ext}\MT@delim@Swapped\of@nostar[\big{]}{x,(s,t)}=\MT@delim@Swapped\of@star{\operatorname{Ext}_{1}(x,s),\operatorname{Ext}_{2}\MT@delim@Swapped\of@nostar[\big{]}{\operatorname{W}_{1}(x,s),t}} is a KL-extractor. Furthermore, if is average-case then so is .
Remark 5.16*.*
The pair is a special case of what Raz and Reingold [RR99] called an extractor-condenser pair. One can think of as preserving “leftovers” or “waste,” which is then “re-extracted” or “recycled” by . The identity function on is a valid choice of , but the advantage of the more general formulation is that can be much smaller than , and most known explicit constructions of extractors have seed length depending on the input length of the source.
Proof.
Given any joint distribution such that is distributed over and , we have for every that
[TABLE]
where the last line follows from the chain rule for KL divergence. Note that
[TABLE]
where the line (*) follows from standard properties of conditional min-entropy (e.g. [DORS08, Lemma 2.2]). That is a KL-extractor now follows immediately from Eq. 5.16.1 by taking independent of , and the average-case claim follows from taking expectations over . ∎
Remark 5.17*.*
The proof above in fact works any weak divergence such that for all joint distributions independent of . In particular, the proof also gives Lemma 5.15 for standard (total variation) extractors.
By Lemma 5.2, we get an analogous result for strong KL-extractors.
Corollary 5.18**.**
Let
* be a strong KL-extractor,* 2. 2.
* be a function such that the map is an injection,* 3. 3.
* be a strong average-case KL-extractor for .*
Then defined by \operatorname{Ext}\MT@delim@Swapped\of@nostar[\big{]}{x,(s,t)}=\MT@delim@Swapped\of@star{\operatorname{Ext}_{1}(x,s),\operatorname{Ext}_{2}\MT@delim@Swapped\of@nostar[\big{]}{\operatorname{W}_{1}(x,s),t}} is a strong KL-extractor. Furthermore, if is average-case then so is .
The zig-zag product for extractors due to Reingold, Vadhan, and Wigderson [RVW00] (in the special case of injective -pairs) is an immediate consequence of Lemma 5.15 and Theorem 5.4 our basic composition result. Recall that Theorem 5.4 was able to combine an “outer” extractor, generally taken to have seed length depending only (but linearly) on , with an “inner” extractor to produce seeds for the outer extractor with logarithmic seed length. However, as discussed in 5.5 that basic composition necessarily lost bits of entropy, so the zig-zag product uses Lemma 5.15 to recover this entropy, using an -pair to ensure that the re-extraction adds additional seed length depending logarithmically on rather than .
Corollary 5.19** **(Zig-zag product for KL-extractors, analogous to
[RVW00, Theorem 3.6]).
Let
* be an extractor for with ,* 2. 2.
* be a function such that the pair is an injection from ,* 3. 3.
* be an average-case KL-extractor,* 4. 4.
* be such that the pair is an injection from ,* 5. 5.
* be an average-case KL-extractor,*
and define
* by \operatorname{Ext}_{comp}\MT@delim@Swapped\of@nostar[\big{]}{(x,y),s}=\operatorname{Ext}_{out}\MT@delim@Swapped\of@nostar[\big{]}{x,\operatorname{Ext}_{in}(y,s)} as in Theorem 5.4,* 2. 2.
* by ,* 3. 3.
* by*
[TABLE]
as in Lemma 5.15.
Then is an -extractor for KL. Furthermore, if and are strong average-case KL-extractors, then is a strong KL-extractor, and if is average-case then so is .
Proof.
We claim that is such that is an injection: by assumption on we have that given and we can recover and , and by assumption on given and we can recover , so that has an inverse and is injective as desired. Therefore, since Theorem 5.4 implies is an KL-extractor, the result follows from Lemma 5.15. The furthermore claims follow from the corresponding claims of these lemmas (and Corollary 5.18 for the strong case). ∎
Remark 5.20*.*
Corollary 5.19 was presented by Reingold, Vadhan, and Wigderson [RVW00] as a transformation that combined three extractor-condenser pairs into a new extractor-condenser pair. We do not use this generality, so for simplicity we do not present it here, but both Lemma 5.15 and Corollary 5.19 can be easily extended in this manner if required.
The Raz–Reingold–Vadhan [RRV02] transformation to avoid entropy loss follows similarly using the Leftover Hash Lemma (5.6).
Corollary 5.21** (KL-extractor analogue of [RRV02, Lemma 28]).**
Let be a strong KL-extractor with entropy loss , meaning . Then for every there is an explicit strong KL-extractor with seed length and entropy loss , meaning , which is computable in polynomial time making one oracle call to . Furthermore, if is average-case then so is .
In particular, by taking we get an extractor with optimal entropy loss by paying an additional in seed length.
Proof.
Let be given by , and let be the strong average-case KL-extractor of 5.6 using almost-universal hash functions, so that and . The result follows from taking to be the extractor of Corollary 5.18. ∎
Remark 5.22*.*
An analogous versions of the above claim for non-strong KL-extractors follows by taking and using Lemma 5.15.
We can apply Corollary 5.21 to Theorem 5.14 the KL-extractors from the total variation extractors of Guruswami, Umans, and Vadhan [GUV09], thereby avoiding the extra entropy loss in the strong extractors.
Corollary 5.23**.**
For every , , and with , there is an explicit strong average-case KL-extractor with and .
5.4 Lower bounds
In this section, we give lower bounds on extractors for the Rényi divergences of all orders, including the special case of KL-extractors. A reader primarily interested in explicit constructions of subgaussian samplers can skip to Section 6.
For Rényi divergences with we reduce to Radhakrishnan and Ta-Shma’s [RT00] lower bounds for total variation extractors and dispersers, which can be understood as a one-sided relaxation of total variation extractors.
Definition 5.24** (Sipser [Sip88], Cohen and Wigderson [CW89]).**
A function is a disperser if for all random variables over with , it holds that .
Dispersers are of interest in the context of Rényi extractors because the Rényi [math]-entropy of a random variable is the logarithm of its support size (see 2.4), and hence dispersers are equivalent to -extractors:
Lemma 5.25**.**
* is a disperser if and only if is a \MT@delim@Swapped\of@star{k,\log\MT@delim@Swapped\of@nostar[\big{]}{1/(1-\varepsilon)}} -extractor.*
Given Lemma 5.25, we can use the Radhakrishnan and Ta-Shma [RT00] lower bounds to give an optimal lower bound on the seed length of -extractors for in terms of the error , input length and supported entropy (we will give a matching non-explicit upper bound in the next section), as well as lower bounds on the entropy loss. For the case of KL-extractors, the non-explicit upper bound (Theorem 5.30) also shows that the entropy loss lower bound is optimal.
Theorem 5.26**.**
Let and be a extractor for with , , and . Then and . Furthermore, if is at most then .
Proof.
Since is nondecreasing in we have that is a extractor for , and thus by Lemma 5.25 it is a disperser. Then the disperser seed length lower bound of Radhakrishnan and Ta-Shma [RT00] tells us that and .
For the other entropy loss lower bound, we use Gilardoni’s [Gil10] generalization of Pinsker’s inequality, which shows in particular that . Thus, is also a total variation extractor, and if (equivalently ) then the [RT00] total variation extractor entropy loss lower bound implies that . ∎
Remark 5.27*.*
For the case of , we do not know whether the entropy loss lower bound of Theorem 5.26 is tight.
It is well-known that -extractors (which are equivalent to -extractors by 2.4) require seed length at least linear in (see e.g. [Vad12, Problem 6.4]). We generalize this to give a linear seed length lower bound on extractors for all , in the regime of constant , improving on the logarithmic lower bound given by Theorem 5.26.
Theorem 5.28**.**
Let be a -extractor for . Then d\geq\min\big{\{}(n-k-3)\cdot\alpha,(m-2)\cdot\alpha/(\alpha+1)\big{\}}.
Proof.
We follow the strategy suggested by Vadhan [Vad12, Problem 6.4], and view as a bipartite graph with left-vertices, right-vertices, and edges per left-vertex given by .
Assume for the sake of contradiction that and , so that and . Now, we claim there exists a set of size at most such that has size at least . This follows from the following iterative procedure: until , choose the vertex of highest degree, add it to , and remove and its neighbors from the graph (the neighbors go in ). Then at each step we will add to a number of vertices at least the average degree
[TABLE]
so that the size of will be at most as desired. Now, since has size at least and is a -extractor, we have that
[TABLE]
which is a contradiction, as desired. ∎
We can also use this lower bound to get a similar lower bound for -extractors for all , though in this case the lower bound applies up to an error threshold that depends on .
Corollary 5.29**.**
Let be a extractor for where and . Then d\geq\min\big{\{}(n-k-3)\cdot\alpha,(m-2)\cdot\alpha/(\alpha+1)\big{\}}.
Proof.
Note that the proof of Theorem 5.28 gave a lower bound on the sum where , whereas . For these can be related without any loss, but in general we can use the triangle inequality to get
[TABLE]
so that if where , then , and we conclude by Lemmas 5.1 and 5.28. ∎
5.5 Non-explicit construction
In this section, we show non-constructively the existence of KL-extractors matching the lower-bound of Theorem 5.26 and in particular implying the optimal parameters of standard extractors for total variation distance. Formally, we will prove:
Theorem 5.30**.**
For every , , and there is an average-case (respectively strong average-case) KL-extractor with seed length and output length (respectively ).
Remark 5.31*.*
For the above parameters are not necessarily optimal, and it would be interested to get matching upper and lower bounds in this regime of parameters.
We will prove Theorem 5.30 using the probabilistic method, analogously to Zuckerman [Zuc97] or Radhakrishnan and Ta-Shma [RT00] for total variation extractors. However, rather than using Hoeffding’s inequality, we use the following lemma:
Lemma 5.32**.**
Let be uniform over a subset of of size . Then if is a random function, it holds for every that
[TABLE]
where and .
Remark 5.33*.*
For total variation extractors, the analogous bound is
[TABLE]
One sees that the bounds are very similar, except the KL divergence version depends on rather than . For the regime where the linear dependence is preferable, and is responsible for the seed length for KL-extractors compared to the seed length for total variation extractors.
Proof of Lemma 5.32.
Note that for each and fixed , the random variable is uniform over the multiset . Hence, since is a random function, this multiset is distributed exactly as taking iid uniform samples from , so we wish to bound the KL divergence between this empirical distribution and the true distribution. For this, the author [Agr19] gave the moment generating function bound
[TABLE]
for every , which for is at most . Then since is independent across , we have
[TABLE]
We can now prove Theorem 5.30:
Proof of Theorem 5.30..
We will show that a random function is a strong average-case KL-extractor with positive probability, the non-strong version then follows from Lemma 5.2. By Lemma 3.14, it is enough to prove that is a strong KL-extractor for every . To reduce the range of we need to consider, note that it suffices to be a extractor for every , so that by rounding down it is enough to be a strong KL-extractor for each such that is an integer.
Now, consider a fixed such that is an integer. Since the KL divergence is convex in its first argument and all distributions of min-entropy at least are convex combinations of “flat” distributions which are uniform over a set of size (Chor and Goldreich [CG88]), it suffices to analyze the behavior of on such distributions. Then for every subset of size , Lemma 5.32 tells us that
[TABLE]
where , , and . There are such subsets of for which we simultaneously need to establish that , so we have by a union bound that the probability that is not a strong average-case KL-extractor is at most
[TABLE]
Hence, as long as
[TABLE]
we know that a random function is a strong average-case KL-extractor with positive probability as desired. ∎
6 Constructions of subgaussian samplers
6.1 Subconstant and
The goal of this section is to establish the following theorem, which is our explicit construction of subgaussian samplers with sample complexity having no dependence on , and with randomness complexity and sample complexity matching the best-known -valued sampler when and are subconstant (up to the hidden polynomial in the sample complexity).
Theorem 6.1**.**
For all , , and there exists an explicit absolute averaging sampler (respectively strong absolute averaging sampler) for subgaussian and subexponential functions with sample complexity and randomness complexity (respectively ).
We will use essentially the same construction used for bounded samplers in this regime, namely applying the Reingold, Wigderson, and Vadhan [RVW00] zig-zag product for extractors to combine the expander extractor of Goldreich and Wigderson [GW97] and an extractor with logarithmic seed length. However, as described in detail in Section 4.1, even the basic composition used in this construction does not work for general subgaussian extractors, so we will instead use the zig-zag product for KL-extractors (Corollary 5.19) combining extractors for Rényi divergences, specifically the -extractor from Corollary 5.11 and the KL-extractor from Corollary 5.23, to get the following high-entropy KL-extractor:
Theorem 6.2**.**
For all integers and there is an explicit average-case (respectively strong average-case) KL-extractor with (respectively ), , and .
Proof.
We prove the claim for strong extractors, for the non-strong claim one can simply define where has length .
By Corollary 5.11, there is a universal constant such that for there is an explicit average-case -extractor with . Furthermore, has the property that the function is such that is an injection.
Let , , and for some parameter to be chosen later. Then by Corollary 5.23, there is an explicit strong average-case KL-extractor with , , and . Furthermore, the function is an injection.
Furthermore, for , by Corollary 5.23 there is also an explicit strong average-case KL-extractor such that and .
Then by the zig-zag product for KL-extractors (Corollary 5.19), there is an explicit strong average-case KL-extractor , where we have
[TABLE]
Choosing so that C\cdot\MT@delim@Swapped\of@nostar[\big{]}{1/(1-\beta)-1}\leq\alpha gives the claim. ∎
We can now prove Theorem 6.1.
Proof of Theorem 6.1.
Let be the explicit \MT@delim@Swapped\of@star{n-\log\MT@delim@Swapped\of@nostar[\big{]}{1/(\delta/2)},\varepsilon^{2}} KL-extractor (respectively strong KL-extractor) of Theorem 6.2, so that and (respectively ).
Then by Lemmas 4.9 and 5.1, is also an \MT@delim@Swapped\of@star{n-\log\MT@delim@Swapped\of@nostar[\big{]}{1/(\delta/2)},\varepsilon} -extractor (respectively strong -extractor), so by Theorem 3.8 the function given by is an explicit sampler for (respectively strong sampler for ), and thus by symmetry of an explicit absolute subexponential sampler (respectively absolute strong subexponential sampler) as desired. ∎
6.2 Constant
We recall from the introduction that the pairwise independent sampler of Chor and Goldreich [CG89] works for subgaussian functions, and in fact the more general class of functions with bounded variance. The sampler has exponentially worse dependence on than is necessary for subgaussian samplers, but is very simple and has randomness complexity optimal up to constant factors.
Theorem 6.3** ([CG89]).**
For all and with , there is an explicit strong sampler for functions with bounded variance , with randomness complexity and sample complexity defined as where is drawn at random from a size pairwise-independent hash family of functions from .
Proof.
The fact that pairwise independence gives rise to a strong bounded-variance sampler is immediate by Chebyshev’s inequality. The existence of a pairwise indepenent hash family with the claimed parameters is due to Chor and Goldreich [CG89], with similar constructions in the probability literature due to Joffe [Jof71]. ∎
We also show that the Expander Neighbor sampler of [KPS85, GW97] is a bounded-variance sampler.
Theorem 6.4**.**
There is a universal constant such that for all and there is an explicit sampler for functions with bounded variance , with randomness complexity and sample complexity . Moreover, if the algorithm is given access to a consistently labelled neighbor function of a Ramanujan graph over of degree , then one can take .
Proof.
By Corollary 5.11, there is an explicit -extractor with , where one can take given the assumed Ramanujan graph. Then by 2.11 is also an -extractor, so we conclude by Theorem 3.8. ∎
Remark 6.5*.*
Note that given explicit constructions of Ramanujan graphs, Theorem 6.4 has the same sample complexity but better randomness complexity than the sampler of Theorem 6.3.
6.3 Non-explicit construction
Applying Lemmas 5.1 and 4.9 to Theorem 5.30 our non-explicit construction of KL-extractors gives:
Corollary 6.6**.**
For every , , and there is an average-case (respectively strong average-case) -extractor with and (respectively )
Since -extractors are also total variation extractors, Corollary 6.6 is optimal up to additive constants by the lower bound of Radhakrishnan and Ta-Shma [RT00].
Using the fact that extractors are samplers (Theorem 3.8), we get
Corollary 6.7**.**
For every integer and there is a sampler (respectively strong sampler) for subgaussian and subexponential functions with sample complexity and randomness complexity (respectively ).
Note that this matches the best-known (non-explicit) parameters of averaging samplers for -valued functions due to Zuckerman [Zuc97].
7 Acknowledgements
The author would like to thank Jarosław Błasiok for suggesting the problem of constructing subgaussian samplers and for helpful discussions and feedback, Salil Vadhan for many helpful discussions and his detailed feedback on this writeup, and the anonymous reviewers for their helpful comments and feedback.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1[AE 05] K. M. R. Audenaert and J. Eisert, “Continuity bounds on the quantum relative entropy,” Journal of Mathematical Physics , vol. 46, no. 10, p. 102104, Oct. 2005.
- 2[Agr 19] R. Agrawal, “Concentration of the multinomial in Kullback–Leibler divergence near the ratio of alphabet and sample sizes,” ar Xiv:1904.02291 [cs, math, stat] , Apr. 2019.
- 3[AS 66] S. M. Ali and S. D. Silvey, “A General Class of Coefficients of Divergence of One Distribution from Another,” Journal of the Royal Statistical Society. Series B (Methodological) , vol. 28, no. 1, pp. 131–142, 1966.
- 4[BBR 88] C. H. Bennett, G. Brassard, and J.-M. Robert, “Privacy Amplification by Public Discussion,” SIAM Journal on Computing , vol. 17, no. 2, pp. 210–229, Apr. 1988.
- 5[BDK + 11] B. Barak, Y. Dodis, H. Krawczyk, O. Pereira, K. Pietrzak, F.-X. Standaert, and Y. Yu, “Leftover Hash Lemma, Revisited,” in Advances in Cryptology – CRYPTO 2011 , ser. Lecture Notes in Computer Science, P. Rogaway, Ed. Springer Berlin Heidelberg, 2011, pp. 1–20.
- 6[BGG 93] M. Bellare, O. Goldreich, and S. Goldwasser, “Randomness in interactive proofs,” computational complexity , vol. 3, no. 4, pp. 319–354, Dec. 1993.
- 7[Bła 18a] J. Błasiok, Private Communication, Cambridge, MA USA, 2018.
- 8[Bła 18b] ——, “Optimal streaming and tracking distinct elements with high probability,” in Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms , ser. Proceedings. Society for Industrial and Applied Mathematics, Jan. 2018, pp. 2432–2448.