Strengthened Information-theoretic Bounds on the Generalization Error

TL;DR

This paper develops new information-theoretic bounds on the probability of error events in learning, improving existing bounds by leveraging various dependence measures like mutual information and maximal leakage.

Contribution

It introduces strengthened bounds on generalization error using multiple information-theoretic metrics, surpassing previous bounds in tightness.

Findings

Mutual information bounds can outperform existing bounds significantly.

New bounds are demonstrated using lautum information, maximal leakage, and $J_ty$.

The bounds provide tighter control over overfitting in learning algorithms.

Abstract

The following problem is considered: given a joint distribution $P_{XY}$ and an event $E$, bound $P_{XY}(E)$ in terms of $P_XP_Y(E)$ (where $P_XP_Y$ is the product of the marginals of $P_{XY}$) and a measure of dependence of $X$ and $Y$. Such bounds have direct applications in the analysis of the generalization error of learning algorithms, where $E$ represents a large error event and the measure of dependence controls the degree of overfitting. Herein, bounds are demonstrated using several information-theoretic metrics, in particular: mutual information, lautum information, maximal leakage, and $J_\infty$. The mutual information bound can outperform comparable bounds in the literature by an arbitrarily large factor.

Equations

$$P(E)\le f_{Q(E)}^{-1}\left(D(P||Q)\right).$$

$$P_{XY}(E)\le f_{P_X{P}_Y(E)}^{-1}\left(I(X;Y)\right).$$

$${\hat{f}}_q(y)=$$

$$\frac{2{\log }^2(2)+\left(\log (1-q)+y\right)\log \frac{(1-q)}{q}+(\log 4)\sqrt{g_q(y)}}{{\log }^2\left((1-q)/q\right)+{\log }^2(2)}.$$

$$P(E)\le {\hat{f}}_{Q(E)}\left(D(P||Q)\right).$$

$$P_{XY}(E)\le {\hat{f}}_{P_X{P}_Y(E)}\left(I(X;Y)\right).$$

$$\underset{q\to 0}{\lim }\underset{D(P||Q)\to 0}{\lim }\frac{{\tilde{f}}_q(D(P||Q))}{{\hat{f}}_q(D(P||Q))}$$

$$\stackrel{(a)}{=}\underset{q\to 0}{\lim }\frac{\left(2{\log }^2(2)+q_1{q}_2+(2\log 2)\sqrt{{\log }^2(2)-q_2\log (q)}\right)\log (1/q)}{\left(q_1^2+{\log }^2(2)\right)\log (2)}$$

$$=\underset{q\to 0}{\lim }\frac{\left(2{\log }^2(2)+{\log }^2(1-q)-\log (q)\log (1-q)+(2\log 2)\sqrt{{\log }^2(2)-\log (1-q)\log (q)}\right)\log (1/q)}{\left({\log }^2(1-q)+{\log }^2(q)-2\log (q)\log (1-q)+{\log }^2(2)\right)\log (2)}$$

$$\stackrel{(b)}{=}\underset{q\to 0}{\lim }\frac{4{\log }^2(2)\log (1/q)}{\left({\log }^2(q)+{\log }^2(2)\right)\log (2)}$$

$$=0,$$

$$\frac{d{\hat{f}}_q(y)}{dy}=\log \frac{1-q}{q}+\log (4)\frac{-2y-\log (q(1-q))}{2\sqrt{g_q(y)}}.$$

$$-\log (1-q)+p\log \left(\frac{1-q}{q}\right)-h(p)\le D(P||Q),$$

$$-$$

$$\frac{d\tilde{g}}{dp}=\log \left(\frac{1-q}{q}\right)-(\log 4)\frac{1-2p}{\sqrt{p(1-p)}}.$$

$$p\log \frac{1-q}{q}-\log \left(1-q\right)-(\log 4)\sqrt{p(1-p)}=y.$$

$$(p{q}_1-q_2-y{)}^2$$

$$p^2\left(q_1^2+{\log }^2(4)\right)$$

$$+(q_2+y{)}^2=0.$$

$$\frac{\Delta }{4}={\left(2{\log }^2(2)+q_1(q_2+y)\right)}^2-(q_1^2+{\log }^2(4))(q_2+y{)}^2$$

$$=(q_2+y)(4q_1{\log }^2(2)-({\log }^2(4))(q_2+y))+4{\log }^4(2)$$

$$=\left(4{\log }^2(2)\right)\left({\log }^2(2)+(q_2+y)(q_1-q_2-y)\right)\ge 0,$$

$$P(E)\le \frac{D(P||Q)+\log (2)}{\log \left(1/Q(E)\right)}.$$

$$D(P||Q)=\underset{\begin{array}{c}f:\mathcal{Z}\to \mathbb{R},{\mathbf{E}}_Q[e^f]<+\infty \end{array}}{\sup }\left\{{\mathbf{E}}_P[f]-\log {\mathbf{E}}_Q[e^f]\right\}.$$

$$P(E)\le \frac{D(P||Q)+\log \left(1+(e^{\beta }-1)Q(E)\right)}{\beta }.$$

$${\ell }^{\star }(t)=\{\begin{array}{ll}0,& 0\le t<Q(E),\\ D(t||Q(E)),& Q(E)\le t\le 1,\\ +\infty ,& t>1.\end{array}$$

$$D(P||Q)$$

$$\ge \beta (P(E)-Q(E))-{\beta }^2/8,$$

$$P(E)\le Q(E)+\sqrt{\frac{D(P||Q)}{2}}.$$

$$P(E)\le 1-e^{-h\left(Q(E)\right)-D(Q||P)}.$$

$$P_{XY}(E)\le 1-e^{-h\left(P_X{P}_Y(E)\right)-L(X;Y)},$$

$$q\log \left(\frac{1-p}{p}\right)-\log (1-p)-h(q)\le D(Q||P).$$

$$D(Q||P)=\underset{\begin{array}{c}f:\mathcal{Z}\to \mathbb{R},{\mathbf{E}}_P[e^f]<+\infty \end{array}}{\sup }\left\{{\mathbf{E}}_Q[f]-\log {\mathbf{E}}_P[e^f]\right\}.$$

$$P(E)\le \frac{1-e^{-D(Q||P)-\beta Q(E)}}{1-e^{-\beta }}.$$

$$P_{XY}(E)\le \alpha \exp \left\{\mathcal{L}\left(X\to Y\right)\right\},$$

Full text

I Introduction

One of the main challenges in designing learning algorithms is guaranteeing that they generalize well [3, 4, 5, 6]. The analysis is made especially hard by the fact that, in order to handle large data sets, learning algorithms are typically adaptive. A recent line of work initiated by Dwork et al. [7, 8, 9] shows that differentially private algorithms provide generalization guarantees. More recently, Russo and Zou [10], and Xu and Raginsky [11], provided an information-theoretic framework for this problem, and showed that the mutual information (between the input and output of the learning algorithm) can be used to bound the generalization error, under a certain assumption. Jiao et al. [12] and Issa and Gastpar [2] relaxed this assumption and provided new bounds using new information-theoretic measures.

The aforementioned papers focus mainly on the expected generalization error. In this paper, we study instead the probability of an undesirable event (e.g., large generalization error in the learning setting). In particular, given an event $E$ and a joint distribution $P_{XY}$, we bound $P_{XY}(E)$ in terms of $P_{X}P_{Y}(E)$ (where $P_{X}P_{Y}$ is the product of the marginals of $P_{XY}$) and a measure of dependence between $X$ and $Y$.

Bassily et al. [13] and Feldman and Steinke [14] provide a bound of this form, where the measure of dependence is mutual information, $I(X;Y)$. We present a new bound in terms of mutual information, which can outperform theirs by an arbitrarily large factor. Moreover, we prove a new bound using lautum information (a measure introduced by Palomar and Verdú [1]). We demonstrate two further bounds using maximal leakage [15, 16] and $J_{\infty}(X;Y)$ (which was recently introduced by Issa and Gastpar [2]). One advantage of the latter two bounds is that they can be computed using only a partial description of the joint $P_{XY}$, hence they are more amenable to analysis.

II Main Results

Let $P_{XY}$ be a joint probability distribution on alphabets ${\mathcal{X}}\times{\mathcal{Y}}$, and let $E\subseteq{\mathcal{X}}\times{\mathcal{Y}}$ be some (“undesirable”) event. We want to bound $P_{XY}(E)$ in terms of $P_{X}P_{Y}(E)$ (where $P_{X}P_{Y}$ is the product of the marginals induced by the joint $P_{XY}$) and a measure of dependence between $X$ and $Y$.

II-A KL Divergence/Mutual Information Bounds

Consider the following intermediate problem: let $P$ and $Q$ be two probability distributions on an alphabet ${\mathcal{Z}}$, and let $E\subseteq{\mathcal{Z}}$ be some event. We will bound $P(E)$ in terms of $Q(E)$ and $D(P||Q)$. Then by replacing $P$ by $P_{XY}$ and $Q$ by $P_{X}P_{Y}$, we get a bound for our desired setup in terms of the mutual information $I(X;Y)=D(P_{XY}||P_{X}P_{Y})$.

Proposition 1

Given $q\in(0,1)$, define $f_{q}:[q,1]\rightarrow\mathbb{R}_{+}$ as $f_{q}(p)=D(p||q)$. Then, $f_{q}(p)$ is a strictly increasing function of $p$. Given any event $E$ and pair of distributions $P$ and $Q$ with $D(P||Q)\leq\log\frac{1}{Q(E)}$,

[TABLE]

In particular, given an event $E\subseteq{\mathcal{X}}\times{\mathcal{Y}}$ and a joint distribution $P_{XY}$ satisfying $I(X;Y)\leq\log\frac{1}{P_{X}P_{Y}(E)}$,

[TABLE]

Proof:

Note that $\frac{df_{q}(p)}{dp}=\log\left(\frac{p}{q}\frac{1-q}{1-p}\right)>0$ for $p>q$, hence $f_{q}(p)$ is strictly increasing. Moreover, the range of $f_{q}(p)$ is $[0,\log(1/q)]$, so (1) is well defined.

If $P(E)\leq Q(E)$, then (1) holds trivially since $f_{Q(E)}^{-1}(D(P||Q))\geq Q(E)$ by the definition of $f$. Otherwise, if $P(E)>Q(E)$, then $f_{Q(E)}\left(P(E)\right)=D\left(P(E)||Q(E)\right)\leq D(P||Q)$, where the second inequality follows from the data processing inequality. Since $f_{q}$ is strictly increasing, then so is $f_{q}^{-1}$. Hence $P(E)\leq f^{-1}_{Q(E)}\left(D(P||Q)\right)$. ∎

The bound above is tight in the following sense. Let $g:[0,1]\times\mathbb{R}_{+}\rightarrow[0,1]$ be such that, given any alphabet ${\mathcal{Z}}$ and event $E\subseteq{\mathcal{Z}}$, and any two distributions $P$ and $Q$ on ${\mathcal{Z}}$, $P(E)\leq g\left(Q(E),D(P||Q)\right)$. Then $g\left(Q(E),D(P||Q)\right)\geq f^{-1}_{Q(E)}\left(D(P||Q)\right)$ if $D(P||Q)\leq\log\frac{1}{Q(E)}$. This is true since given any tuple $({\mathcal{Z}},P,Q,E)$ such that $D(P||Q)\leq\log\frac{1}{Q(E)}$, there exists $({\mathcal{Z}}^{\prime},P^{\prime},Q^{\prime},E^{\prime})$ such that $Q(E^{\prime})=Q(E)$, $D(P||Q)=D(P^{\prime}||Q^{\prime})$, and (1) holds with equality. In particular, choose ${\mathcal{Z}}^{\prime}=\{0,1\}$, $E^{\prime}=\{1\}$, $Q^{\prime}\sim\mathrm{Ber}(Q(E))$, and $P^{\prime}\sim\mathrm{Ber}\left(f_{Q(E)}^{-1}\left(D(P||Q)\right)\right)$.

However, there is no closed form for the bound in (1). The following corollary provides an upper bound in closed form:

Corollary 1

Given $q\in(0,1/2]$, define $g_{q}(y):=\log^{2}(2)+\left(\log(1-q)+y\right)\left(-\log(q)-y\right)$ and $\hat{f}_{q}:[0,-\log(q))\rightarrow\mathbb{R}_{+}$ as follows:

[TABLE]

Then, $\hat{f}_{q}(y)$ is concave and non-decreasing in $y$. Moreover, given any event $E$ and pair of distributions $P$ and $Q$ with $D(P||Q)\leq\log\frac{1}{Q(E)}$,

[TABLE]

In particular, given an event $E\subseteq{\mathcal{X}}\times{\mathcal{Y}}$ and a joint $P_{XY}$ satisfying $I(X;Y)\leq\log\frac{1}{P_{X}P_{Y}(E)}$,

[TABLE]

Proof:

Since $g_{q}(y)$ is concave in $y$ and the square root is concave and non-decreasing, $\sqrt{g_{q}(y)}$ is concave in $y$; hence $\hat{f}_{q}(y)$ is concave in $y$. To show that it is non-decreasing, consider the derivative (ignoring the positive denominator):

[TABLE]

For $y\in[0,-\frac{1}{2}\log(q(1-q))]$, both terms are non-negative (the first is non-negative since $q\leq 1/2$). For $y\in[-\frac{1}{2}\log(q(1-q)),-\log(q)]$, the numerator of the second term is negative and decreasing, and the denominator is positive and decreasing. Hence, it achieves its minimum for $y=-\log(q)$. Since the minimum \frac{d\hat{f}_{q}(y)}{dy}\bigg{|}_{y=-\log(q)}=0, we get that $\frac{d\hat{f}_{q}(y)}{dy}\geq 0$ for $y\in[0,-\log(q)]$.

Now, let $p:=P(E)$ and $q:=Q(E)$. Then we can rewrite the inequality $D(p||q)\leq D(P||Q)$ as

[TABLE]

where $h(.)$ is the binary entropy function (in nats). Upper-bounding $h(p)\leq(\log 4)\sqrt{p(1-p)}$ , we get

[TABLE]

For ease of notation, let $y:=D(P||Q)$ and $\tilde{g}(p)$ be the left-hand side. Then,

[TABLE]

Hence, there exists $p_{0}$ such that $\tilde{g}$ is decreasing on $[0,p_{0}]$ and increasing on $[p_{0},1]$. Therefore, $\tilde{g}(p)=y$ admits at most two solutions, say $p_{1}<p_{2}$, and $\tilde{g}(p)\leq y\Rightarrow p\leq p_{2}$. It remains to solve

[TABLE]

Let $q_{1}=\log\frac{1-q}{q}$, and $q_{2}=\log(1-q)$. We get

[TABLE]

The discriminant of (9) is given by

[TABLE]

where the inequality follows from the fact that $q_{1}-q_{2}-y=-\log(q)-y\geq 0$. Hence, the larger root of (9) is given by $\hat{f}_{q}(p)$, as desired. ∎

II-A1 Comparison with existing bound

It has been shown [14, Lemma 3.11][13, Lemma 9] that

[TABLE]

The bound in Corollary 1 can be arbitrarily smaller than (10). That is, let $\tilde{f}_{Q(E)}(D(P||Q))$ be the left-hand side of (10) and consider the calculation shown at the top of the next page.

Moreover, one can derive a family of bounds in the form of (10) using the Donsker-Varadhan characterization of KL. In particular,

[TABLE]

Now, let $f=\beta\mathbb{I}\{z\in E\}$ for some $\beta>0$, where $\mathbb{I}\{\}$ is the indicator function. After rearranging terms, we get

[TABLE]

Choosing $\beta=\log(1/Q(E))$, we slightly improve (10) by replacing $\log(2)$ with $\log(2-Q(E))$. In fact, we can solve the infimum over $\beta>0$ of the right-hand side of (12). In particular, by [17, Lemma 2.4], the infimum is given by $\ell^{\star-1}(D(P||Q))$, where $\ell^{\star}$ is the convex conjugate of111Lemma 2.4 of [17] assumes $\ell^{\prime\prime}(0)=0$, but the proof goes as is for $\ell^{\prime\prime}(0)\geq 0$, which is the case here. $\ell(\beta)=\log(1+(e^{\beta}-1)Q(E))$, and $\ell^{\star-1}(y)=\inf\{t:\ell^{\star}(t)>y\}$. It turns out that $\ell^{\star}:\mathbb{R}_{+}\rightarrow\mathbb{R}_{+}$ is given by

[TABLE]

Now, $P(E)\leq\inf\{t:\ell^{\star}(t)>D(P||Q)\}$. Hence, for $D(P||Q)=0$, $P(E)\leq\inf(Q(E),+\infty)=Q(E)$. By noting that $\ell^{\star}(1)=\log(1/Q(E))$, we get for any $D(P||Q)>\log(1/Q(E))$, $P(E)\leq\inf(1,+\infty)=1$. Finally, for $D(P||Q)\in(0,\log(1/Q(E))]$, we get $P(E)\leq\{t\in[Q(E),1]:D(t||Q(E))>D(P||Q)\}$, which is equal to $t^{\star}\in[Q(E),1]$ satisfying $D(t^{\star}||Q(E))=D(P||Q)$. That is, the bound derived from (13) exactly recovers Proposition 1.

Furthermore, we could compare with the “mutual information bound” of Russo and Zou [10], and Xu and Raginsky [11]. In particular, by considering $f=\beta\left(\mathbb{I}\{z\in E\}-Q(E)\right)$ for $\beta\in\mathbb{R}$ in (11), we get

[TABLE]

where the second inequality follows from the fact that $(\mathrm{Ber}(q)-q)$ is $\frac{1}{4}$-subgaussian (which is true for any random variable whose support has length 1). Since the above inequality holds for any $\beta\in\mathbb{R}$, we get

[TABLE]

Evidently, Corollary 1 can outperform (14) since (for finite $D(P||Q)$) $\lim_{Q(E)\rightarrow 0}\hat{f}_{Q(E)}\left(D(P||Q)\right)=0$, whereas the right-hand side of (14) goes to $\sqrt{D(P||Q)/2}$. In Figure 1, we plot the 3 bounds (equations (6), (10), and (14)) for a given range of interest: small $Q(E)$, and relatively small $D(P||Q)$, e.g., proportional to $-\log(1-Q(E))$.

Remark 1

Given the form of the 3 bounds, one might expect that (14) outperforms the other two for large values of $D(P||Q)$. This is in fact not true because the range of interest for the right-hand sides is restricted to $[0,1]$. For instance, for small $Q(E)$ and $D(P||Q)=-\log(Q(E))/2$, the bound in (14) is trivial ($>1$), and the other two bounds are strictly less than 1.

II-A2 Bound using $D(Q||P)$/Lautum Information

By considering the data processing inequality $D(q||p)\leq D(Q||P)$, we can bound $p$ in terms of $q$ and $D(Q||P)$.

Theorem 1

Given any event $E$ and a pair of distributions $P$ and $Q$, if $P(E)\leq 1/2$, then

[TABLE]

In particular, given an event $E\subseteq{\mathcal{X}}\times{\mathcal{Y}}$ and a joint distribution $P_{XY}$ with $P_{XY}(E)\leq 1/2$,

[TABLE]

where $L(X;Y):=D(P_{X}P_{Y}||P_{XY})$ is the lautum information [1].

Proof:

Set $p=P(E)$ and $q=Q(E)$. As in (6), we can rewrite $D(q||p)\leq D(Q||P)$ as

[TABLE]

Since $p\leq 1/2$ (by assumption), we can drop the first term of the left-hand side. Rearranging the inequality then yields Theorem 1. ∎

Moreover, we can derive a family of bounds similar to (12) by considering the Donsker-Varadhan representation of $D(Q||P)$:

[TABLE]

Now, let $f=-\beta\mathbb{I}\{z\in E\}$ for some $\beta>0$. Then after rearranging terms, we get for any $\beta>0$,

[TABLE]

II-B Maximal Leakage Bound

The bounds presented so far in (4) and (15) do not take into account the specific relation of $P_{XY}$ and $P_{X}P_{Y}$ as a joint distribution and its marginal. Indeed, they are applications of a more general bound that can be applied to an arbitrary pair of distributions (Corollary 1 and Theorem 1). The following bound does not fall under this category, i.e., it only applies to pairs of distributions forming a joint and marginal.

Theorem 2

Given $\alpha\in[0,1]$, finite alphabets ${\mathcal{X}}$ and ${\mathcal{Y}}$, a joint distribution $P_{XY}$ and an event $E\subseteq{\mathcal{X}}\times{\mathcal{Y}}$ such that for all $y\in{\mathcal{Y}}$, $P_{X}(E_{y})\leq\alpha$ where $E_{y}:=\{x:(x,y)\in E\}$, then

[TABLE]

where $\mathcal{L}\left(X\!\!\to\!\!Y\right)=\log\sum_{y\in{\mathcal{Y}}}\max_{\begin{subarray}{c}x:\\ P_{X}(x)>0\end{subarray}}P_{Y|X}(y|x)$ is the maximal leakage.

Remark 2

The bound holds more generally but we restrict our attention to finite alphabets to make the presentation of the proof simple.

Proof: Fix $y\in{\mathcal{Y}}$ satisfying $P_{Y}(y)>0$, and consider the pair of distributions $P_{X|Y=y}$ and $P_{X}$:

[TABLE]

where the equalities follow from [18, Theorem 6]. Hence,

[TABLE]

Now,

[TABLE]

where (a) follows from the following (readily verifiable) facts:

[TABLE]

One advantage of the bound of Theorem 2 is that it depends on a partial description of $P_{Y|X}$ only. Hence, it is simpler to analyze than the mutual information bounds. Moreover, for fixed $P_{X}$, the bound is convex in $P_{Y|X}$. In the next subsection, we present a bound with similar properties.

II-C $J_{\infty}$-Bound

Theorem 3

Given $\alpha\in[0,1/2]$, finite alphabets ${\mathcal{X}}$ and ${\mathcal{Y}}$, a joint distribution $P_{XY}$ and an event $E\subseteq{\mathcal{X}}\times{\mathcal{Y}}$ such that for all $y\in{\mathcal{Y}}$, $P_{X}(E_{y})\leq\alpha$ where $E_{y}:=\{x:(x,y)\in E\}$, then

[TABLE]

where $J_{\infty}(X;Y)=\frac{1}{2}\sum_{y\in{\mathcal{Y}}}\left(\max_{x}P_{Y|X}(y|x)-\min_{x}P_{Y|X}(y|x)\right)$ [2].

Proof:

The theorem follows from Theorem 1 and Corollary 1 of [2]. In particular, following the same proof steps as in [2], one can show that for any function222In [2], the authors consider $X=(X_{1},\cdots,{\mathcal{X}}_{n})$, ${\mathcal{Y}}=\{1,2,\cdots,n\}$, and $f(X,Y)=X_{Y}$. Nevertheless, the proof of (21) remains the same. $f:{\mathcal{X}}\times{\mathcal{Y}}\rightarrow\mathbb{R}$,

[TABLE]

where $\mu_{y}:={\rm\mathbf{E}}_{P_{X}}[f(X,y)]$. Now, set $f(x,y)=\mathbb{I}\{(x,y)\in E\}$. Then, ${\rm\mathbf{E}}_{P_{XY}}[f(X,Y)]=P_{XY}(E)$, ${\rm\mathbf{E}}_{P_{X}P_{Y}}[f(X,Y)]=P_{X}P_{Y}(E)\leq\alpha$, and ${\rm\mathbf{E}}_{P_{X}}[f(X,y)]=P_{X}(E_{y})$. Moreover,

[TABLE]

where the last inequality follows from the assumption that $P_{X}(E_{y})\leq\alpha\leq\frac{1}{2}$. Then, it follows from (21) that

[TABLE]

The theorem follows by noting that $P_{X}P_{Y}(E)\leq\alpha$. ∎