A Constructive Proof of a Concentration Bound for Real-Valued Random Variables

TL;DR

This paper extends a constructive proof of a concentration bound originally for Boolean variables to real-valued variables, improving the bound and providing an efficient algorithm for identifying dependent variables.

Contribution

It generalizes the Impagliazzo-Kabanets algorithm to real-valued variables and enhances the concentration bound by a constant factor.

Findings

Extended the constructive proof to real-valued variables.

Improved the concentration bound by a constant factor.

Provided an efficient randomized algorithm for dependency detection.

Abstract

Almost 10 years ago, Impagliazzo and Kabanets (2010) gave a new combinatorial proof of Chernoff's bound for sums of bounded independent random variables. Unlike previous methods, their proof is constructive. This means that it provides an efficient randomized algorithm for the following task: given a set of Boolean random variables whose sum is not concentrated around its expectation, find a subset of statistically dependent variables. However, the algorithm of Impagliazzo and Kabanets (2010) is given only for the Boolean case. On the other hand, the general proof technique works also for real-valued random variables, even though for this case, Impagliazzo and Kabanets (2010) obtain a concentration bound that is slightly suboptimal. Herein, we revisit both these issues and show that it is relatively easy to extend the Impagliazzo-Kabanets algorithm to real-valued random variables and to improve the corresponding concentration bound by a constant factor.

Equations

$$\mathbb{E}\Big{[}\prod_{i\in S}X_{i}\Big{]}\leq\prod_{i\in S}c_{i},$$

$$\mathbb{P}\big{(}X\geq(c+t)n\big{)}\leq e^{-D\big{(}\frac{c-a+t}{b}\,\big{\|}\,\frac{c-a}{b}\big{)}n}.$$

$$\mathbb{P}(\mathcal{I}=S)={\lambda }^{|S|}(1-\lambda {)}^{n-|S|},$$

$$\mathbb{E}\Big{[}\prod_{i\in S}\widetilde{X}_{i}\Big{]}\leq\prod_{i\in S}\tilde{c}_{i}.$$

$$\displaystyle\mathbb{E}\Big{[}\prod_{i\in S}\widetilde{X}_{i}\Big{]}$$

$$\displaystyle=\frac{1}{b^{|S|}}\sum_{\mathcal{I}\subseteq S}\Big{(}\prod_{j\in S\setminus\mathcal{I}}(-a_{j})\Big{)}\,\mathbb{E}\Big{[}\prod_{i\in\mathcal{I}}X_{i}\Big{]}$$

$$\displaystyle\leq\frac{1}{b^{|S|}}\sum_{\mathcal{I}\subseteq S}\Big{(}\prod_{j\in S\setminus\mathcal{I}}(-a_{j})\Big{)}\Big{(}\prod_{i\in\mathcal{I}}c_{i}\Big{)}$$

$$=\frac{1}{b^{|S|}}\prod _{i\in S}(c_i-a_i)=\prod _{i\in S}{\tilde{c}}_i,$$

$$\mathbb{E}\Big{[}\prod_{i\in S}Y_{i}\Big{]}\leq\prod_{i\in S}\tilde{c}_{i}.$$

$$\displaystyle\mathbb{E}\Big{[}\prod_{i\in S}Y_{i}\Big{]}$$

$$\displaystyle=\mathbb{E}\Bigg{[}\prod_{i\in S}\mathbb{E}\big{[}Y_{i}\,\big{|}\,\widetilde{X}_{1},\dots,\widetilde{X}_{n}\big{]}\Bigg{]}$$

$$\displaystyle=\mathbb{E}\Big{[}\prod_{i\in S}\widetilde{X}_{i}\Big{]}\leq\prod_{i\in S}\tilde{c}_{i},$$

$$\mathbb{E}\Big{[}\prod_{i\in\mathcal{I}}Y_{i}\Big{]}\leq\big{(}\lambda\tilde{c}+1-\lambda\big{)}^{n}.$$

$$\displaystyle\mathbb{E}\Big{[}\prod_{i\in\mathcal{I}}Y_{i}\Big{]}$$

$$\le \sum _{S\subseteq [n]}{\lambda }^{|S|}(1-\lambda {)}^{n-|S|}\prod _{i\in S}{\tilde{c}}_i$$

$$\displaystyle=\sum_{S\subseteq[n]}\Big{(}\prod_{i\in S}\lambda\tilde{c}_{i}\Big{)}\Big{(}\prod_{i\in[n]\setminus S}(1-\lambda)\Big{)}$$

$$\displaystyle=\prod_{i=1}^{n}\big{(}\lambda\tilde{c}_{i}+1-\lambda\big{)}$$

$$\displaystyle\leq\left(\frac{1}{n}\sum_{i=1}^{n}\big{(}\lambda\tilde{c}_{i}+1-\lambda\big{)}\right)^{n}$$

$$\displaystyle=\big{(}\lambda\tilde{c}+1-\lambda\big{)}^{n},$$

$$\mathbb{E}\Big{[}\prod_{i\in\mathcal{I}}Y_{i}\,\Big{|}\,X\geq(c+t)n\Big{]}\geq(1-\lambda)^{n-(\tilde{c}+\tilde{t})n}.$$

$$(1-\lambda {)}^{1-x}=1-(1-x)\lambda +\sum _{i=2}^{\infty }(\genfrac{}{}{0ex}{}{1-x}{i})(-\lambda {)}^i\le 1-(1-x)\lambda ,$$

$$\displaystyle\phantom{=}\mathbb{E}\Bigg{[}\prod_{i\in\mathcal{I}}Y_{i}\,\Big{|}\,X\geq(c+t)n\Bigg{]}$$

$$\displaystyle=\mathbb{E}\Bigg{[}\mathbb{E}\Big{[}\prod_{i\in\mathcal{I}}Y_{i}\,\Big{|}\,X_{1},\dots,X_{n}\Big{]}\,\Bigg{|}\,X\geq(c+t)n\Bigg{]}$$

$$\displaystyle=\mathbb{E}\Bigg{[}\prod_{i\in\mathcal{I}}\widetilde{X}_{i}\,\Big{|}\,X\geq(c+t)n\Bigg{]}$$

$$\displaystyle=\sum_{S\subseteq[n]}\mathbb{P}\big{(}\mathcal{I}=S\big{)}\mathbb{E}\Big{[}\prod_{i\in S}\widetilde{X}_{i}\,\Big{|}\,X\geq(c+t)n\Big{]}$$

$$\displaystyle=\mathbb{E}\Bigg{[}\,\sum_{S\subseteq[n]}\mathbb{P}\big{(}\mathcal{I}=S\big{)}\prod_{i\in S}\widetilde{X}_{i}\,\Big{|}\,X\geq(c+t)n\Bigg{]}$$

$$\displaystyle=\mathbb{E}\Bigg{[}\sum_{S\subseteq[n]}\Big{(}\prod_{i\in S}\lambda\widetilde{X}_{i}\Big{)}\Big{(}\prod_{i\in[n]\setminus S}(1-\lambda)\Big{)}\,\Big{|}\,X\geq(c+t)n\Bigg{]}$$

$$\displaystyle=\mathbb{E}\Bigg{[}\prod_{i=1}^{n}\big{(}\lambda\widetilde{X}_{i}+1-\lambda\big{)}\,\Big{|}\,X\geq(c+t)n\Bigg{]}$$

$$\displaystyle\geq\mathbb{E}\Bigg{[}\prod_{i=1}^{n}(1-\lambda)^{1-\widetilde{X}_{i}}\,\Big{|}\,X\geq(c+t)n\Bigg{]}$$

$$\displaystyle=\mathbb{E}\Bigg{[}(1-\lambda)^{n-\frac{X-na}{b}}\,\Big{|}\,X\geq(c+t)n\Bigg{]}\geq(1-\lambda)^{n-(\tilde{c}+\tilde{t})n},$$

$$(\lambda \tilde{c}+1-\lambda {)}^n$$

$$\displaystyle\geq\mathbb{E}\Big{[}\prod_{i\in\mathcal{I}}Y_{i}\,\Big{|}\,X\geq(c+t)n\Big{]}\,\mathbb{P}\big{(}X\geq(c+t)n\big{)}$$

$$\displaystyle\geq(1-\lambda)^{n-(\tilde{c}+\tilde{t})n}\,\mathbb{P}\big{(}X\geq(c+t)n\big{)},$$

$$\mathbb{P}\big{(}X\geq(c+t)n\big{)}\leq\Bigg{(}\frac{\lambda\tilde{c}+1-\lambda}{(1-\lambda)^{1-\tilde{c}-\tilde{t}}}\Bigg{)}^{n}.$$

$$g(\lambda_{*})=\Bigg{(}\frac{\tilde{c}}{\tilde{c}+\tilde{t}}\Bigg{)}^{\tilde{c}+\tilde{t}}\Bigg{(}\frac{1-\tilde{c}}{1-\tilde{c}-\tilde{t}}\Bigg{)}^{1-\tilde{c}-\tilde{t}}=e^{-D\big{(}\tilde{c}+\tilde{t}\,\big{\|}\,\tilde{c}\big{)}}.$$

$$\displaystyle\mathbb{P}\big{(}X\geq(c+t)n\big{)}$$

$$\displaystyle\leq\mathbb{E}\Big{[}\prod_{i=1}^{n}\widetilde{X}_{i}\Big{]}\leq\prod_{i=1}^{n}\tilde{c}_{i}$$

$$\displaystyle\leq\Big{(}\frac{1}{n}\sum_{i=1}^{n}\tilde{c}_{i}\Big{)}^{n}=\tilde{c}^{n}=e^{-D(1\,\|\,\tilde{c})n}.$$

$$\mathbb{P}\big{(}X\geq(c+t)n\big{)}=p>2\alpha,$$

$$\mathbb{E}\Big{[}\prod_{i\in S}X_{i}\Big{]}>c^{\left|S\right|}+\Omega\big{(}\alpha^{4/ct}\big{)}.$$

$$\mathbb{E}\Big{[}\prod_{i\in\mathcal{I}}Y_{i}\Big{]}\geq\mathbb{P}\big{(}X\geq(c+t)n\big{)}\mathbb{E}\Big{[}\prod_{i\in\mathcal{I}}Y_{i}\,\Big{|}\,X\geq(c+t)n\Big{]}\geq p\,(1-\lambda)^{n(1-c-t)}.$$

Full text

Herein, we revisit both these issues and show that it is relatively easy to extend the Impagliazzo-Kabanets algorithm to real-valued random variables and to improve the corresponding concentration bound by a constant factor.

generalized Chernoff-Hoeffding bound,

concentration bound,

randomized algorithm.,

[**]aCorresponding author

]Freie Universität Berlin

1 Introduction

The weak law of large numbers is a central pillar of modern probability theory: any sample average of independent random variables converges in probability to its expected value. This qualitative statement is made more precise by concentration bounds, which quantify the speed of convergence for certain prominent special cases. Due to their wide applicability in mathematics, statistics, and computer science, a whole industry of concentration bounds has developed over the last decades. By now, the literature contains myriads of different bounds, satisfying various needs and proved in numerous ways; see, e.g., Chernoff (1952), Hoeffding (1963), Schmidt et al. (1995), Panconesi and Srinivasan (1997), or the interesting and extensive textbooks and surveys by McDiarmid (1998), Chung and Lu (2006), Alon and Spencer (2008), and Mulzer (2018).

In theoretical computer science, a central application area of concentration bounds lies in the design and analysis of randomized algorithms. About 10 years ago, Impagliazzo and Kabanets (2010) went in the other direction, showing that methods from theoretical computer science can be useful in obtaining new proofs for concentration bounds. This led to a new—algorithmic—proof of a generalized Chernoff bound for Boolean random variables, which the authors “consider more revealing and intuitive than the standard Bernstein-style proofs, and hope that its constructiveness will have other applications in computer science”.

Impagliazzo and Kabanets were able to extend their combinatorial approach to real-valued bounded random variables. However, in order to do this, they had to use a slightly different argument. This came at the cost of a sub-optimal multiplicative constant in the bound. Furthermore, with the new argument, it was not clear how to generalize the main randomized algorithm to the real-valued case. Here, we present a constructive proof of Chernoff’s bound that remedies both these issues, giving the same bound and the same algorithmic result as are known for the Boolean case.

2 The Real-Valued Impagliazzo-Kabanets Theorem

The main result of Impagliazzo and Kabanets for real-valued bounded random variables is stated as Theorem 2.1 below. Essentially, this theorem can be seen as a very simple adaptation of the famous Chernoff-Hoeffding bound (Shenkman, 2018, Computation (4.6)). In their paper, Impagliazzo and Kabanets (2010) proved the bound with a sub-optimal multiplicative constant. Very recently, Pelekis and Ramon (2017) claimed the bound with an optimal constant, but, unfortunately, their argument is flawed (Shenkman, 2018, Remark 5.10). We present a new proof of Theorem 2.1 that leads to an optimal multiplicative constant and a randomized algorithm for real-valued bounded random variables. Remarkably, our result is obtained by following the original approach of Impagliazzo and Kabanets for the Boolean case, and pushing through the calculation to the end thanks to Lemma 2.6.

In what follows, we set $[n]:=\{1,\dots,n\}$, for $n\in\mathbb{N}$, and use $\mathbb{E}[\cdot]$ for the expectation operator. Moreover, for $p,q\in[0,1]$, we denote by $D\left(p\,\|\,q\right):=p\ln(p/q)+(1-p)\ln((1-p)/(1-q))$ the binary relative entropy with the conventions $0\ln 0=0$ and $\ln(x/0)=\infty$, for all $x\in(0,1]$.

Theorem 2.1**.**

Suppose we are given a sequence $X_{1},\dots,X_{n}$ of $n$ random variables, and $2n+1$ real constants $a_{1},\dots,a_{n},b,c_{1},\dots,c_{n}$ such that $a_{1},\dots,a_{n}\leq 0$, $b>0$, and $a_{i}\leq X_{i}\leq a_{i}+b$ almost surely, for all $i\in[n]$, and

[TABLE]

for all $S\subseteq[n]$. Set $X:=\sum_{i=1}^{n}X_{i}$, $a:=(1/n)\sum_{i=1}^{n}a_{i}$, and $c:=(1/n)\sum_{i=1}^{n}c_{i}$. Then, for any $t\in[0,b+a-c]$, we have

[TABLE]

Remark 2.2**.**

The condition $a_{i}\leq 0$, for $i\in[n]$, in Theorem 2.1 can be overcome by imposing a stronger condition of dependence than (2.1); see (Shenkman, 2018, Theorem 3.11).

The remainder of this section is dedicated to the proof of Theorem 2.1, which follows the ideas found in Impagliazzo and Kabanets (2010). We first deal with the case where $c\in(a,b+a)$ and $t<b+a-c$. We fix a parameter $\lambda\in[0,1)$, and we consider the following random process: for $i=1,\dots,n$, we sample the random variable $X_{i}$. Then, we normalize the resulting values to obtain a sequence $\widetilde{X}_{1},\dots,\widetilde{X}_{n}$ of probabilities. We use these probabilities to sample $n$ conditionally independent Boolean random variables $Y_{1},\dots,Y_{n}$. Finally, we go through the $Y_{i}$, and for each $i$, we set $Y_{i}$ to $1$ with probability $1-\lambda$ and we keep it unchanged with probability $\lambda$. Now, the aim is to bound the expected value \mathbb{E}\big{[}\prod_{i=1}^{n}Y_{i}\big{]} in two different ways. On the one hand, it will turn out that (2.1) implies that \mathbb{E}\big{[}\prod_{i=1}^{n}Y_{i}\big{]} can be upper-bounded by $(\lambda\tilde{c}+1-\lambda)^{n}$, the expectation of the product of $n$ independent Boolean random variables that are set to $1$ with probability $\lambda\tilde{c}+1-\lambda$, where $\tilde{c}$ is the normalized average of the $c_{i}$. On the other hand, the expectation \mathbb{E}\big{[}\prod_{i=1}^{n}Y_{i}\big{]} can be lower-bounded by \mathbb{P}\big{(}X\geq(c+t)n\big{)} times the conditional expectation given the event $X\geq(c+t)n$, which turns out to be at least $(1-\lambda)^{n-(\tilde{c}+\tilde{t})n}$, where $\tilde{t}$ is the normalized deviation parameter $t$. Combining the two bounds and optimizing for $\lambda$ will then lead to (2.2).

We now proceed with the details. For $i\in[n]$, we define the normalized variables $\widetilde{X}_{i}:=(X_{i}-a_{i})/b$ and the normalized constants $\tilde{c}_{i}:=(c_{i}-a_{i})/b$, as well as $\tilde{c}:=(c-a)/b$ and $\tilde{t}:=t/b$. We define $n$ Boolean random variables Y_{i}\sim\text{Bernoulli}\big{(}\widetilde{X}_{i}\big{)}, for $i\in[n]$, that are conditionally independent given $\widetilde{X}_{1},\dots,\widetilde{X}_{n}$. In other words, $Y_{1},\dots,Y_{n}$ are independent on the $\sigma$-algebra generated by the set \big{\{}\widetilde{X}_{i}\mid i\in[n]\big{\}}. Furthermore, let $\lambda\in[0,1)$ be fixed, and let $\mathcal{I}$ be a random variable, independent of $\widetilde{X}_{1},\dots,\widetilde{X}_{n}$ and of $Y_{1},\dots,Y_{n}$, taking values in $\{S\mid S\subseteq[n]\}$ with the probability mass function

[TABLE]

for all $S\subseteq[n]$. If $\mathbb{P}(X\geq(c+t)n)=0$, then (2.2) holds trivially, so from now on we assume that $\mathbb{P}(X\geq(c+t)n)>0$. As mentioned, our goal is to bound the expectation \mathbb{E}\big{[}\prod_{i\in\mathcal{I}}Y_{i}\big{]} in two different ways, and we start with the upper bound. The first lemma shows that the condition (2.1) on the moments of the $X_{i}$ carries over to the normalized variables $\widetilde{X}_{i}$.

Lemma 2.3**.**

For any $S\subseteq[n]$, we have

[TABLE]

Proof.

The lemma follows quickly by plugging in the definitions. More precisely, we have

[TABLE]

as claimed. ∎

The normalized moment condition from Lemma 2.3 now shows that expectation of $\prod_{i\in S}Y_{i}$ can be upper-bounded by the expectation of a product of independent Boolean random variables with success probabilities $\tilde{c}_{i}$.

Lemma 2.4**.**

For any $S\subseteq[n]$, we have

[TABLE]

Proof.

The lemma follows by the conditional independence of the $Y_{i}$. We have

[TABLE]

as desired. ∎

To achieve an upper bound for \mathbb{E}\big{[}\prod_{i\in\mathcal{I}}Y_{i}\big{]}, we must still account for the random subset $\mathcal{I}$. Essentially, it says that we can think of the expected product of $n$ independent Boolean random variables with success probability $\lambda\tilde{c}+1-\lambda$.

Lemma 2.5**.**

We have

[TABLE]

Proof.

We proceed as follows:

[TABLE]

as stated. ∎

We turn to the lower bound for \mathbb{E}\big{[}\prod_{i\in\mathcal{I}}Y_{i}\big{]}. For this, we first bound the conditional expectation given the event $X\geq(c+t)n$ assuming that \mathbb{P}\big{(}X\geq(c+t)n\big{)}>0.

Lemma 2.6**.**

We have

[TABLE]

Proof.

First, we note that for $\lambda\in[0,1)$ and $x\in[0,1]$, the binomial series expansion gives

[TABLE]

since $\binom{1-x}{i}=\frac{\prod_{j=0}^{i-1}(1-x-j)}{i!}$, and thus $\sum_{i=2}^{\infty}\binom{1-x}{i}(-\lambda)^{i}\leq 0$. The derivation proceeds as follows:

[TABLE]

as desired. ∎

Now, combining Lemmas 2.5 and 2.6 with the law of total expectation, we obtain that for any $\lambda\in[0,1)$,

[TABLE]

and hence, for any $\lambda\in[0,1)$,

[TABLE]

A straightforward calculation shows that $g(\lambda):=(\lambda\tilde{c}+1-\lambda)/(1-\lambda)^{1-\tilde{c}-\tilde{t}}$ is minimized at $\lambda_{*}:=\tilde{t}/((1-\tilde{c})(\tilde{c}+\tilde{t}))\in[0,1)$ and that

[TABLE]

To complete the proof, it remains to consider the cases where $c=a$ or $t=b+a-c$. First, observe that $c=a$ implies $c_{i}=a_{i}$, for all $i\in[n]$, which, in turn, gives $X_{i}=a_{i}$ almost surely for all $i\in[n]$. Consequently, we have \mathbb{P}\big{(}X\geq(a+t)n\big{)}=1=e^{-D(0\,\|\,0)n}, if $t=0$, and \mathbb{P}\big{(}X\geq(a+t)n\big{)}=0=e^{-D(\tilde{t}\,\|\,0)n}, if $t>0$. Second, if $c>a$ and $t=b+a-c$, we have that

[TABLE]

This concludes the proof of Theorem 2.1.

3 Algorithmic Implications

We provide a generalization of Theorem $4.1$ in Impagliazzo and Kabanets (2010).

Theorem 3.1**.**

There is a randomized algorithm $\mathcal{A}$ such that the following holds. Let $X_{1},\dots,X_{n}$ be $[0,1]$-valued random variables. Let $0<c<1$ and $0<t\leq 1-c$ be such that

[TABLE]

for some $\alpha\geq e^{-D\left(c+t\,\|\,c\right)n}$. Then, on inputs $n,c,t,\alpha$, the algorithm $\mathcal{A}$, using oracle access to the distribution of $(X_{1},\dots,X_{n})$, runs in time $\text{poly}(\alpha^{-1/ct},n)$ and outputs a set $S\subseteq[n]$ such that, with probability at least $1-o(1)$, one has

[TABLE]

Proof.

We follow the argument of Impagliazzo and Kabanets: for $i\in[n]$, let the random variables $Y_{i}\sim\text{Bernoulli}(X_{i})$ be conditionally independent given the $X_{i}$. Furthermore, for $\lambda\in(0,1)$, let $\mathcal{I}\sim\text{Binomial}(n,\lambda)$ be a random set that is independent of the $X_{i}$ and of the $Y_{i}$. Using the law of total expectation and Lemma 2.6, we infer that

[TABLE]

Moreover, by proceeding as in the proof of Lemma 2.5, we can show that \mathbb{E}\big{[}c^{|\mathcal{I}|}\big{]}\leq(\lambda c+1-\lambda)^{n}. Hence, we obtain

[TABLE]

The rest of the proof is completely analogous to (Impagliazzo and Kabanets, 2010, Theorem 4.1). ∎