Strong Chain Rules for Min-Entropy under Few Bits Spoiled

TL;DR

This paper demonstrates that with minimal additional information, sequences of random variables exhibit near-flat conditional distributions, enabling strong chain rules and other properties for min-entropy that are not generally valid, with applications to extractors.

Contribution

It introduces a method to spoil few bits to induce near-flat conditional distributions, enabling strong chain rules and properties for min-entropy.

Findings

Conditional distributions become nearly flat after spoiling few bits.

Reproof of Nisan and Zuckermann's entropy preservation result.

Additive loss of entropy is negligible with enough samples.

Abstract

It is well established that the notion of min-entropy fails to satisfy the \emph{chain rule} of the form $H(X,Y) = H(X|Y)+H(Y)$, known for Shannon Entropy. Such a property would help to analyze how min-entropy is split among smaller blocks. Problems of this kind arise for example when constructing extractors and dispersers. We show that any sequence of variables exhibits a very strong strong block-source structure (conditional distributions of blocks are nearly flat) when we \emph{spoil few correlated bits}. This implies, conditioned on the spoiled bits, that \emph{splitting-recombination properties} hold. In particular, we have many nice properties that min-entropy doesn't obey in general, for example strong chain rules, "information can't hurt" inequalities, equivalences of average and worst-case conditional entropy definitions and others. Quantitatively, for any sequence $X_1,\ldots,X_t$ of random variables over an alphabet $\mathcal{X}$ we prove that, when conditioned on $m = t\cdot O( \log\log|\mathcal{X}| + \log\log(1/\epsilon) + \log t)$ bits of auxiliary information, all conditional distributions of the form $X_i|X_{<i}$ are $\epsilon$-close to be nearly flat (only a constant factor away). The argument is combinatorial (based on simplex coverings). This result may be used as a generic tool for \emph{exhibiting block-source structures}. We demonstrate this by reproving the fundamental converter due to Nisan and Zuckermann (\emph{J. Computer and System Sciences, 1996}), which shows that sampling blocks from a min-entropy source roughly preserves the entropy rate. Our bound implies, only by straightforward chain rules, an additive loss of $o(1)$ (for sufficiently many samples), which qualitatively meets the first tighter analysis of this problem due to Vadhan (\emph{CRYPTO'03}), obtained by large deviation techniques.

Equations

$$H_1(X|Y)=H(X,Y)-H(Y).$$

$$er{r}_{\mathsf{Spoil}}$$

$$er{r}_{\mathsf{Samp}}$$

$$d_{TV}(X;Y)=\frac{1}{2}\sum _{x\in \mathcal{X}}|\mathrm{Pr}[X=x]-\mathrm{Pr}[Y=x]|$$

$$H_1(X)=-\sum _x\mathrm{Pr}[X=x]\log \mathrm{Pr}[X=x].$$

$$H_{\infty }(X)=-\log \underset{x\in \mathcal{X}}{\max }\mathrm{Pr}[X=x].$$

$$H_{\infty }(X|Y)=\underset{y\in \mathcal{Y}}{\min }{H}_{\infty }(X|Y=y).$$

$${\tilde{H}}_{\infty }(X|Y)=-\log \left({\mathbb{E}}_{y\leftarrow Y}{2}^{-H_{\infty }(X|Y=y)}\right).$$

$$H_{\infty }^{ϵ}(X)=\underset{X^{\prime }:d_{TV}(X,X^{\prime })⩽ϵ}{\max }{H}_{\infty }(X^{\prime })$$

$$d_{TV}(\mathsf{Ext}(X,U_d),U_d;U_m,U_d)⩽ϵ$$

$$\underset{i_1,\dots ,i_{\ell }\leftarrow \mathsf{Samp}(U_r)}{\mathrm{Pr}}\left[\frac{1}{\ell }\sum _{i=1}^{\ell }f(i)⩾\mu -\theta \right]⩽1-\gamma .$$

$$\log |\mathcal{B}|=t\cdot O\left(\log \log |\mathcal{X}|+\log \log ({ϵ}^{-1})+\log (t/\delta )\right)$$

$$\sum _{B\in \mathcal{B}}{p}_X(B)⩾1-ϵ$$

$$\forall x,x^{\prime }\in B:2^{-O(\delta )}⩽\frac{p_X(x_i|x_{<i})}{p_X(x_i^{\prime }|x_{<i}^{\prime })}⩽2^{O(\delta )}.$$

$$H_{\infty }(X_i|X_I,B)=H_{\infty }(X_i,X_I|B)-H_{\infty }(X_I|B)\pm O(\delta ).$$

$${\tilde{H}}_{\infty }(X_i|X_I,B)=H_{\infty }(X_i|X_I,B)\pm O(\delta ).$$

$$\mathrm{Pr}[X_{⩽i}=x_{⩽i}|B]=2^{\pm O(\delta )}\mathrm{Pr}[X_{<i}=x_{<i}|B].$$

$$\mathrm{Pr}[X_i=x^{\prime \prime },X_I=x_I^{\prime }|B]=2^{\pm O(\delta )}\mathrm{Pr}[X_I=x_I^{\prime }|B].$$

$$H_{\infty }(X_i|X_I,B)=H_{\infty }(X_i,X_I,B)-H_{\infty }(X_I,B)\pm O(\delta )$$

$${\tilde{H}}_{\infty }(X_i|X_I,B)=H_{\infty }(X_i|X_I,B)\pm O(\delta )$$

$$p_i(x_i|x_{i-1},\dots ,x_1)=p_{X_i|X_{<i}}(x_i,\dots ,x_1).$$

$$r^i(x)=-\log p_i(x_i,\dots ,x_1).$$

$$\sum _{i=1}^t{r}^i(x)=-\log p(x).$$

$$\forall x\in A:-\log p(x_1,\dots ,x_n)⩽t\log |\mathcal{X}|+\log (1/ϵ).$$

$$\forall x\in B:\left|r^i(x)-z_i\right|⩽R$$

$$\forall x,x^{\prime }\in B:\left|r^i(x)-r^i(x^{\prime })\right|⩽2R,i=1,\dots ,t.$$

$$r_B^i(x)$$

$$=-\log p_B(x_i|x_{i-1},\dots ,x_1)$$

$$\forall x,x^{\prime }\in B:\left|r_B^i(x)-r_B^i(x^{\prime })\right|⩽2R,i=1,\dots ,t.$$

$$\begin{gathered} \frac{1}{\ell \log |\mathcal{X}|}{\tilde{H}}_{\infty }^{ϵ}(X_{i_{\ell }}{X}_{i_{\ell -1}}{X}_{i_{\ell -2}}\dots ,X_{i_1}|\mathcal{B})⩾ \\ \frac{1}{t\log |\mathcal{X}|}{\tilde{H}}_{\infty }(X_{i_{\ell }}{X}_{i_{\ell -1}}{X}_{i_{\ell -2}}\dots ,X_{i_1})-er{r}_{\mathsf{Spoil}}-er{r}_{\mathsf{Sampler}} \end{gathered}$$

$$er{r}_{\mathsf{Spoil}}$$

$$er{r}_{\mathsf{Samp}}$$

Full text

We show that any sequence of variables exhibits a very strong strong block-source structure (conditional distributions of blocks are nearly flat) when we spoil few correlated bits. This implies, conditioned on the spoiled bits, that splitting-recombination properties hold. In particular, we have many nice properties that min-entropy doesn’t obey in general, for example strong chain rules, “information can’t hurt” inequalities, equivalences of average and worst-case conditional entropy definitions and others.

Quantitatively, for any sequence $X_{1},\ldots,X_{t}$ of random variables over an alphabet $\mathcal{X}$ we prove that, when conditioned on $m=t\cdot O(\log\log|\mathcal{X}|+\log\log(1/\epsilon)+\log t)$ bits of auxiliary information, all conditional distributions of the form $X_{i}|X_{<i}$ are $\epsilon$-close to be nearly flat (only a constant factor away). The argument is combinatorial (based on simplex coverings).

This result may be used as a generic tool for exhibiting block-source structures. We demonstrate this by reproving the fundamental converter due to Nisan and Zuckermann (J. Computer and System Sciences, 1996), which shows that sampling blocks from a min-entropy source roughly preserves the entropy rate. Our bound implies, only by straightforward chain rules, an additive loss of $o(1)$ (for sufficiently many samples), which qualitatively meets the first tighter analysis of this problem due to Vadhan (CRYPTO’03), obtained by large deviation techniques.

I Introduction

I-A Strong vs Weak Entropy Chain Rules

One of the most useful properties of Shannon entropy is the chain rule, showing how entropy splits between distributions

[TABLE]

The notion of min-entropy, very important for cryptography [Sha11], fails to satisfy this property [IS13]. In the lack of a chain rule, a much weaker one-sided bound (e.g. of the form $\widetilde{H}_{\infty}(X|Y)\geqslant H_{\infty}(X)-H_{0}(Y)$, where $\tilde{H}_{\infty}$ is an appropriate extension of min-entropy to conditional distributions) is sometimes used [DORS08], which we address as a weak chain rule111In leakage-resilient cryptography such bounds are simply called chain rules. In this paper we discuss chain rules in a strong sense..

I-B Need for Strong Chain Rules

While the weak chain rule suffices for many applications related to bounded leakage  [DORS08, DP08] (where $Y$ is leakage much shorter than the amount of min-entropy in $X$), it is insufficient where one needs to estimate how entropy is distributed among blocks. In these settings, one would like to argue that (roughly) either $X$ or $Y|X$ has high min-entropy if the joint min-entropy (of $(X,Y)$) is high. Examples of such problems are randomness extraction in the bounded storage model [Cac97], constructions of dispersers [BRSW06], or oblivious transfer protocols [DFRSS07].

I-C Our Contribution and Related Works

Although the chain rule fails in general, we show that it is true conditioned on few spoiled bits. We actually show more, that (locally, conditioned on auxiliary bits) a very strong block-source structure exists. Namely, each block is nearly flat given previous blocks. Informally, the theorem reads as follows

Theorem** (Informal: exhibiting flat block-source structures).**

For any sequence $X=X_{1},\ldots,X_{t}$ of correlated random variables each over $\mathcal{X}$ and any $\epsilon$ there exists auxiliary information $S$ of length $m$ bits such that

1. (a)

$S$* is short: $m=t\cdot O(\log\log|\mathcal{X}|+\log\log(1/\epsilon)+\log t)$* 2. (b)

Conditioned on $S$, conditional block distributions $\Pr[X_{i}|X_{i}=x_{i},X_{i-1}=x_{i-1},\ldots,X_{1}=x_{1}]$ are nearly flat ($\epsilon$-close to a probability distribution whose values differ by a constant factor).

The formal statement is given in Theorem 1. For cryptographic applications, $\log\log(\epsilon^{-1})$ is pretty much a small constant (typically $\epsilon=2^{-100}$). Also, for sources with super-logarithmic entropy per block, that is when $H_{\infty}(X)\gg t\log\log|\mathcal{X}|)$, and the number of blocks $t$ growing not too fast, e.g. $t=\log^{O(1)}(|\mathcal{X}|)$, the error term is of a smaller order than the entropy. Under these mild assumptions, conditioned on the partition generated by auxiliary bits, we conclude many nice properties that fail in general. Examples are chain rules, “conditioning only decreases entropy” properties, equivalences of conditional entropy defined in different ways and others.

I-C1 Our Tools

Spoiling Knowledge

The spoiling knowledge technique is essentially about finding auxiliary information that increases entropy, and was introduced in [BBCM95]. We use the same idea to force block distributions to be nearly flat.

Covering techniques (combinatorial geometry)

In order to construct a good “spoiling”, we consider the logarithm of the chaining identity $p_{X_{1}\ldots X_{t}}(\cdot)=\prod_{j=1}^{t}p_{X_{i}|X_{i-1}\ldots X_{1}}(\cdot)$ which represents the “surprise” of the total distribution as a sum of “next-block surprises” of the form $r_{i}(\cdot)=-\log p_{X_{i}|X_{i-1}\ldots X_{1}}(\cdot)$. It follows that the vector of all $r_{i}$ (for $i=1,\ldots,t$) lie in a $(t-1)$-dimensional simplex of edge roughly $O(t\log|\mathcal{X}|)$. Our partition is obtained from coverings, as for all $x$ in the same part $r_{i}(\cdot)$ is roughly constant (when the radius is small enough). This approach can be likely optimized (we use a crude bound on the covering number). Also the lower bounds on the necessary number of spoiled buts are possible, by considering packings instead of coverings. We defer this discussion to the full version.

I-D Related works

As far as we know, the presented result on spoiling min-entropy is knew. The Nisan-Zuckerman lemma, discussed in this paper as an application, was analyzed by Vadhan [Vad03] and recently by Bellare [BKR16]. These results study min-entropy present in random chunks of a larger source, and don’t offer tools for splitting entropy in any source deterministically (particularly for a small number of blocks), as we do.

I-E Applications

The important result due to Nisan and Zuckerman [NZ96], improved later by Vadhan [Vad03] states that sampling from a given source of high min-entropy rate $\alpha$ yields a source of a comparable entropy rate $\beta$. This fact is a crucial step in constructions of so called local extractors, that extract randomness parsing only a part of input. For a while, only a lossy bound $\beta\approx\alpha/\log(1/\alpha)$ was known. The reason was precisely the lack of a chain rule for min-entropy. As observed by Cachin [Cac97] the proof for Shannon entropy (a less interesting case) is straightforward and follows basically by a splitting-recombining argument, which uses a chain rule in both directions. We demonstrate by our technique that (surprisingly) a very effective splitting-recombining approach actually works, and achieves $\beta=\alpha-o(1)$ in a very straightforward way. This matches the bound due to Vadhan. Concretely, if the original source is a sequence of $t$ blocks over an alphabet $\mathcal{X}$ and we take $\ell$ samples, then $\beta=\alpha-err_{\mathsf{Spoil}}-err_{\mathsf{Samp}}$ where the losses due to chain rules and sampling equal, respectively

[TABLE]

In particular $\beta$ converges to $\alpha$ when the block length is $\log|\mathcal{X}|=\log^{\omega(1)}(1/\epsilon)$ and $\log|\mathcal{X}|=\omega(1)\cdot t$. For more details see Theorem 2 in Section V. For the discussed result our bounds converge slightly slower than Vadhan’s bounds derived by large deviation techniques. However, our spoiling technique can be used also for small number of samples.

I-F Organization

In Section II we explain necessary notions and notations. Auxiliary facts that will be needed are discussed in Section III. In Section IV we prove the main result. Applications to the bounded storage model are discussed in Section V. We conclude the work in Section VI.

II Preliminaries

II-A Basic Notation

For any random variables $X_{1},X_{2}$ by $p_{X_{1}|X_{2}}$ we denote the distribution of $X_{1}$ conditioned on $X_{2}$, that is $p_{X_{1}|X_{2}}(x_{1},x_{2})=\Pr[X_{1}=x_{1}|X_{2}=x_{2}]$. Throughout this paper, all logarithms are taken to base $2$. For any sequence of random variables $X=X_{1},\ldots,X_{n}$ we denote $X_{<i}=X_{1},\ldots,X_{i-1}$, $X_{\leqslant i}=X_{1},\ldots,X_{i}$ and more generally for any subset $I\subset\{1,\ldots,n\}$ we put $X_{I}=X_{i_{1}}X_{i_{2}}\ldots X_{i_{m-1}}X_{i_{m}}$ where $i_{1}<i_{2}<\ldots<i_{m}$ are all elements of $I$.

II-B Distances, Entropies

In the definitions below $\mathcal{X}$ is an arbitrary finite set.

Definition 1** (Statistical Distance).**

For two random variables $X,Y$ on $\mathcal{X}$ by the statistical distance (total variation) we mean

[TABLE]

Definition 2** (Shannon Entropy).**

The Shannon entropy of a random variable $X$ on $\mathcal{X}$ equals

[TABLE]

Definition 3** (Min-Entropy).**

The min-entropy of a random variable $X$ on $\mathcal{X}$ equals

[TABLE]

Definition 4** (Conditional Min-Entropy [DORS08]).**

Let $X,Y$ be random variables over $\mathcal{X}$ and $\mathcal{Y}$ respectively. The worst-case min-entropy of $X$ conditioned on $Y$ equals

[TABLE]

The average min-entropy of $X$ conditioned on $Y$ equals

[TABLE]

Remark 1**.**

The averaged notion is slightly weaker, but has better properties and actually better suits applications [DORS08].

The notion of smooth entropy is more accurate than min-entropy because quantifies entropy up to small perturbations in the probability mass.

Definition 5** (Smooth Min-Entropy [RW04]).**

The $\epsilon$-smooth min-entropy of a random variable $X$ on $\mathcal{X}$ is defined as

[TABLE]

where the maximum is over all random variables $X^{\prime}$ on $\mathcal{X}$. In other words, $X$ has at least $k$ bits of smooth min-entropy if there is $X^{\prime}$ of min-entropy at least $k$ and $\epsilon$-close to $X$.

II-C Randomness Extractors

Below we recall the definition of seeded extractors

Definition 6** (Randomness Extractor [NZ96]).**

We say that a function $\mathsf{Ext}:\{0,1\}^{n}\times\{0,1\}^{d}\rightarrow\{0,1\}^{m}$ is a $(k,\epsilon)$-extractor if and only if

[TABLE]

for any $X$ on $\{0,1\}^{n}$ with min-entropy at least $k$.

II-D Samplers

Averaging samplers are procedures which sample points, within a given domain, that are distributed enough random to approximate every function. It turns out that there exist good averaging samples, using much less auxiliary randomness than necessary to generate independent points. For our applications we consider samplers that approximate averages from below.

Definition 7** (Averaging Samplers [Vad03]).**

A function $\mathsf{Samp}:\{0,1\}^{d}\rightarrow[t]^{\ell}$ is a $(\mu,\theta,\gamma)$-averaging sampler if for every function $f:[t]\rightarrow[0,1]$ with average value $\frac{1}{\ell}\sum_{i=1}^{\ell}f(i)\geqslant\mu$ it holds that

[TABLE]

Lemma 1** (Optimal Averaging Samplers (Nonconstructive) [Vad03]).**

For the setting in Definition 7 there is an averaging sampler which for any $\ell^{\prime}$ such that $\Omega(\mu\theta^{-2}\log(1/\gamma))\leqslant\ell^{\prime}\leqslant\ell$ produces $\ell^{\prime}$ distinct samples and uses $d=\log(t/\ell^{\prime})+\log(1/\gamma)+2\log(\mu/\theta)+\log\log(1/\mu)+O(1)$ random bits.

III Auxiliary Lemmas

The following lemma is essentially the information can’t hurt principle, well known for Shannon entropy, stated for the notion of min-entropy. It says that extra information can only reduce the uncertainty, and comes up often in cryptography.

Lemma 2** (Conditioning only reduces entropy [IS13]).**

For any $X,Y,Z$ we have $\widetilde{H}_{\infty}(X|Y,Z)\leqslant\widetilde{H}_{\infty}(X|Y)$.

It is well known (see, e.g., [BST03]) that when each block has certain min-entropy entropy conditioned on previous blocks, the total entropy grows linearly with the number of blocks (as expected). The lemma is stated without a proof (it follows directly from definitions). We stress, however, that using the worst-case notion of conditional entropy is crucial.

Lemma 3** (Min-entropy from block sources).**

For any $X,Y,Z$ we have ${H}_{\infty}(X,Y|Z)\geqslant{H}_{\infty}(X|Y,Z)+{H}_{\infty}(Y|Z)$.

Another lemma, well known in folklore, quantifies the intuition that conditioning on “large” events (not too surprising information) doesn’t decrease entropy too much.

Lemma 4** (Leakage lemma for min-entropy).**

For any random variable $X$ and any events $A,B$ we have $H_{\infty}(X|B,A)\geqslant H_{\infty}(X|B)-\log(1/\Pr(A|B))$.

Proof.

The first part of the lemma follows because we have $\Pr[X=x|A,B]=\Pr[X=x,A\cap B]/\Pr[A\cap B]\leqslant\Pr[X=x,B]/(\Pr[B]\cdot\Pr[A|B])$ for every $x$. ∎

IV Main Result

IV-A Chain Rule

Theorem 1** (Exhibiting block structures with few bits spoiled).**

Let $\mathcal{X}$ be a fixed alphabet and $X=(X_{1},\ldots,X_{t})$ be a sequence of (possibly correlated) random variables each over $\mathcal{X}$. Then for any $1>\epsilon>0$ and $\delta>0$ there exists a collection $\mathcal{B}$ of disjoint sets on $\mathcal{X}^{t}$ such that

1. (a)

$\mathcal{B}$* can be indexed by a small number of bits, namely*

[TABLE] 2. (b)

$\mathcal{B}$* almost covers the domain*

[TABLE] 3. (c)

Conditioned on members of $\mathcal{B}$, block distributions $X_{i}|X_{<i}$ are nearly flat.

[TABLE]

for every $B\in\mathcal{B}$ and $i=1,\ldots,t$.

In some applications (see for example Section V) it is convenient to work with parts that are not too small. By substituting $\epsilon:=\frac{\epsilon}{2}$, and deleting from $\mathcal{B}$ all members of smaller than $\frac{\epsilon}{2|\mathcal{B}|}$ (their mass is at most $\frac{\epsilon}{2}$), we obtain

Remark 2** (Getting rid of tiny parts).**

In Theorem 1, we may assume $p_{X}(B)=\Omega\left(\epsilon\cdot|\mathcal{B}|^{-1}\right)$ for every $B\in\mathcal{B}$.

Corollary 1** (Conditional entropies under few bits spoiled).**

Under the assumptions of Theorem 1, for every $B\in\mathcal{B}$ for every index $i$ and for every set $I\subset\{1,\ldots,i-1\}$ we have

1. (a)

The chain rule for min-entropy

[TABLE] 2. (b)

The average and worst-case min-entropy almost match

[TABLE]

Proof of Corollary 1.

Fix any subset $I\subset\{1,\ldots,i-1\}$ of size $m$. By Theorem 1 for every $x$ and $B$ we have

[TABLE]

Let $J=\{1,\ldots,i-1\}\setminus I$. Taking the sum of Equation 2 over $x$ such that $x_{I}=x^{\prime}_{I}$ and $x_{i}=x^{\prime\prime}$ are fixed but $x_{J}$ varies we obtain

[TABLE]

This implies

[TABLE]

which finishes the proof. ∎

Proof of Theorem 1.

Let $p$ be the joint distribution of $X_{1},\ldots,X_{n}$. For any $i$ denote

[TABLE]

and let the “surprise” of the bit $x_{i}$ be

[TABLE]

Note that $p(x_{1},\ldots,x_{n})=\prod_{i=1}^{n}p_{i}(x_{i},x_{i-1},\ldots,x_{1})$ and therefore, denoting $x=(x_{1},\ldots,x_{n})$, we obtain

Claim 1** (Decomposing surprises).**

We have

[TABLE]

The next claim follows by a simple Markov-type argument.

Claim 2** (Significant probabilities).**

There exists a set $A\subset\mathcal{X}^{t}$ of probability $1-\epsilon$ such that $p(x)\geqslant\frac{\epsilon}{|\mathcal{X}|^{t}}$ for all $x\in A$.

Denoting $x=x_{1},\ldots,x_{n}$, we have

[TABLE]

The claim below follows from Equation 3 and Equation 4

Claim 3** (Surprises live in a simplex).**

We have $r^{i}(x)\geqslant 0$ for $i=1,\ldots,t$ and $\sum_{i=1^{t}}r^{i}(x)\leqslant t\log|\mathcal{X}|+\log(1/\epsilon)$, for all points $x\in A$.

Claim 4** (Simplex coverings imply a chain rule).**

If the simplex with side length $t\log|\mathcal{X}|+\log(1/\epsilon)$ can be covered by $N$ balls of radius $R$ in the $\ell_{\infty}$ norm, then the theorem holds with $|\mathcal{B}|=N$ and $\delta=R$.

Proof of 4.

Let $C\subset\mathbb{R}^{t}$, $|C|=N$, be the set of the centers of the covering balls. Let $S$ be the function which assigns to every point $x\in A$ (where $A$ is defined in 2) the point $z\in C$ closest to the vector $(r_{1}(x),\ldots,r_{t}(x))$ in the $\ell_{\infty}$-norm. Fix any $z$ and let $B_{z}=\{x:S(x)=z\}$. By the properties of the covering, for $i=1,\ldots,t$ we obtain

[TABLE]

In particular, the surprises for any two points in $B$ are close

[TABLE]

Let $p_{B}$ be the conditional probability of $p$ given $B$. Denote by $r^{i}_{B}$ the surprise of the $i$-th bit given previous bits and conditioned on $B$, that is

[TABLE]

Note that $p_{B}(x)=p(x)/p(B)$ for $x\in B$, and hence $r^{i}_{B}(x)=r^{i}(x)+\log(p(B)^{-1})$. Now Equation 7 implies

[TABLE]

which finishes the proof. ∎

It remains to observe that the covering number for our case is $\log N=\log N_{0}+\log\log(|\mathcal{X}|)+\log\log(1/\epsilon))+\log(t/\delta)$ (see for example [Reg07]) which finishes the proof. ∎

V Applications

Theorem 2** (Sampling preserves entropy rate [Vad03]).**

Let $\mathcal{X}$ be a fixed finite alphabet, and let $X_{1},\ldots,X_{t}$ be a sequence of correlated random variables each over $\mathcal{X}$. Let $i_{1},\ldots,i_{\ell}\in[1,t]$, where $\ell<t$, be chosen from the set $\{1,\ldots,t\}$ by an averaging $(\mu,\theta,\gamma)$-sampler. Then there is a random variable $\mathcal{B}$ taking $m=t\cdot O(\log\log|\mathcal{X}|+\log\log(1/\epsilon)+\log t)$ bits, such that

[TABLE]

where the errors due to spoiling and sampling equal

[TABLE]

Remark 3** (Local extractors).**

Composing this with an extractor over $\mathcal{X}^{\ell}$ one obtains a local extractor, which reads only a small fraction (specifically $\frac{\ell}{t}$) of input bits. We refer to [Vad03] for a general discussion.

Proof.

We will argue that the sequence $X_{i_{1}},\ldots,X_{i_{\ell}}$, for $\ell$ sufficiently big, likely has the same entropy rate (entropy per block) as the original sequence $X_{1},\ldots,X_{\ell}$.

Let $\mathcal{B}$ be the family guaranteed by Theorem 1. By part (a) of Corollary 1 applied $t$ times (starting from $i=t$ downto $i=1$) we have for every $B\in\mathcal{B}$

[TABLE]

and now by part (b) applied to each summand

[TABLE]

Note that

[TABLE]

In particular, with high probability over $(i_{j})_{j=1,\ldots,\ell}$

[TABLE]

For the sake of clarity, we comment later on the exact error in accuracy and probability in Equation 10. Observe that by part (b) of Corollary 1 we obtain

[TABLE]

(which is the conditioning reduces entropy property). Again, by applying part (b) of Corollary 1 to the sum on the right-hand side of Equation 11 we get

[TABLE]

By Lemma 3 from Equation 11 we get for every $B^{\prime}\in\mathcal{B}^{\prime}$

[TABLE]

Combining this with Equation 11,Equation 10 and Equation 9 we finally obtain (with high probability)

[TABLE]

Note that this holds for every $B$. Recall that by Remark 2 we can assume $\Pr[B]=\Omega(|\mathcal{B}|^{-1}\epsilon)$. Now by Lemma 4 we have $H_{\infty}(X|B)>H_{\infty}(X)-O(\log|\mathcal{B}|+\log(1/\epsilon))$ and thus

[TABLE]

We can do slightly better. Namely, from Equation 14

[TABLE]

where $\mathcal{B}$ is a random variable that assigns to every point $x$ the corresponding set $B$ covering $x$, conditioned in addition on the map being defined (it fails when $x$ is not covered by any $B$ which happens w.p. at most $\epsilon$) Indeed, we have

[TABLE]

and Equation 16 follows by averaging over $B$ and the Jensen Inequality (note that $\ell<t$ implies that the corresponding mapping is concave). By Lemma 3 applied to $X|\mathcal{B}=B$ for all possible $B$ to outcomes of $\mathcal{B}$ we obtain

[TABLE]

It remains to use an explicit bound on $|\mathcal{B}|$ from Theorem 1, set the sampler to $\gamma=\epsilon$ and compute $\theta$ from $\epsilon$ and $\ell$. ∎

VI Conclusion

By a simple combinatorial argument combined with the spoiling knowledge technique we showed how to exhibits strong block-source structures in any min-entropy source. This approach may be applied to locally enforce chain rules (or other desired properties) for min-entropy.