Understanding the Behaviour of the Empirical Cross-Entropy Beyond the Training Distribution

TL;DR

This paper investigates how the empirical cross-entropy behaves when the test data distribution differs from the training data, providing bounds based on mutual information and supporting the theory with numerical simulations.

Contribution

It introduces a theoretical framework linking the testing gap of empirical cross-entropy to mutual information, extending understanding beyond training distribution.

Findings

Testing gap bounded by mutual information between inputs and representations

Numerical simulations confirm mutual information as a predictor of testing gap

Results applicable to understanding generalization beyond training distribution

Abstract

Machine learning theory has mostly focused on generalization to samples from the same distribution as the training data. Whereas a better understanding of generalization beyond the training distribution where the observed distribution changes is also fundamentally important to achieve a more powerful form of generalization. In this paper, we attempt to study through the lens of information measures how a particular architecture behaves when the true probability law of the samples is potentially different at training and testing times. Our main result is that the testing gap between the empirical cross-entropy and its statistical expectation (measured with respect to the testing probability law) can be bounded with high probability by the mutual information between the input testing samples and the corresponding representations, generated by the encoder obtained at training time. These results of theoretical nature are supported by numerical simulations showing that the mentioned mutual information is representative of the testing gap, capturing qualitatively the dynamic in terms of the hyperparameters of the network.

Equations

$$Q_{\hat{Y}|X}(y|x)={\mathbb{E}}_{q_{U|X}}\left[Q_{\hat{Y}|U}(y|U)|X=x\right],$$

$$\ell(x,y)\coloneqq\ell\big{(}q_{U|X}(\cdot|x),{Q}_{\hat{Y}|U}(y|\cdot)\big{)}=\mathbb{E}_{q_{U|X}}\left[-\log Q_{\hat{Y}|U}(y|U)|X=x\right].$$

$${\mathcal{L}}_{\text{emp}}(q_{U|X},Q_{\hat{Y}|U},{\mathcal{S}}_n):=\frac{1}{n}\sum _{i=1}^n\ell (x_i,y_i).$$

$${\mathcal{E}}_{\text{gap}}(q_{U|X},Q_{\hat{Y}|U},{\mathcal{S}}_n):=\left|{\mathcal{L}}_{\text{emp}}(q_{U|X},Q_{\hat{Y}|U},{\mathcal{S}}_n)-\mathcal{L}(q_{U|X},Q_{\hat{Y}|U})\right|,$$

$${\mathcal{E}}_{\text{gap}}(q_{U|X},Q_{\hat{Y}|U},{\mathcal{S}}_n)$$

$$+\frac{D_{\delta }\cdot {\mathcal{D}}_{\text{HL}}\left(Q_{Y|U}^D\Vert Q_{\hat{Y}|U}|q_U^D\right)+C_{\delta }}{\sqrt{n}}+\mathcal{O}\left(\frac{\log (n)}{n}\right),$$

$${\mathcal{D}}_{\text{HL}}\left(Q_{Y|U}^D\Vert Q_{\hat{Y}|U}|q_U^D\right)=\sqrt{\frac{1}{2}\cdot {\mathbb{E}}_{q_U^D}\left[\sum _{y\in \mathcal{Y}}{\left(\sqrt{Q_{\hat{Y}|U}(y|U)}-\sqrt{Q_{Y|U}^D(y|U)}\right)}^2\right]},$$

$$ϵ(K)=\underset{\begin{array}{c}k,x,y:\\ 1\le k\le K\\ y\in \mathcal{Y}\\ x\in {\mathcal{K}}_k^{(y)}\end{array}}{\sup }\left|\ell (x,y)-\ell (x^{(k,y)},y)\right|,r(K)=\frac{1}{\underset{\begin{array}{c}k,y:\\ 1\le k\le K\\ y\in \mathcal{Y}\end{array}}{\min }{\int }_{{\mathcal{K}}_k^{(y)}}{p}_X(x)dx}.$$

$$q_U^D(u)=\sum _{k=1}^K\sum _{y\in \mathcal{Y}}{q}_{U|X}(u|x^{(k,y)}){\int }_{{\mathcal{K}}_k^{(y)}}{p}_{XY}(x,y)dx,$$

$$Q_{Y|U}^D(y|u)=\frac{{\sum }_{k=1}^K{q}_{U|X}(u|x^{(k,y)}){\int }_{{\mathcal{K}}_k^{(y)}}{p}_{XY}(x,y)dx}{q_U^D(u)}$$

$$\sqrt{\mathcal{I}\left(p_{\mathbf{X}};q_{\mathbf{U}|\mathbf{X}}\right)}\leq\sqrt{\sum_{j=1}^{m}\mathcal{D}\left(q_{U_{j}|\mathbf{X}}(\cdot|\mathbf{X})\big{\|}\tilde{q}_{U_{j}}\big{|}p_{\mathbf{X}}\right)}.\!\!$$

$$\mathcal{D}\left(q_{U_{j}|\mathbf{X}}(\cdot|\mathbf{X})\big{\|}\tilde{q}_{U_{j}}\big{|}\hat{P}_{\mathbf{X}}\right)=\frac{1}{2n}\sum_{i=1}^{n}\left(-\log\sigma^{2}_{j}(\mathbf{x}_{i})+\sigma_{j}^{2}(\mathbf{x}_{i})+\mu_{j}^{2}(\mathbf{x}_{i})-1\right).$$

$$\displaystyle\mathcal{D}\left(q_{U_{j}|\mathbf{X}}(\cdot|\mathbf{X})\big{\|}\tilde{q}_{U_{j}}\big{|}\hat{P}_{\mathbf{X}}\right)$$

$$=\frac{1}{n}\sum _{i=1}^n\frac{{\alpha }_j^2({\mathbf{x}}_i)+(\log (f_j({\mathbf{x}}_i))-{\mu }_j{)}^2}{2{\sigma }_j^2}-\log \frac{{\alpha }_j({\mathbf{x}}_i)}{{\sigma }_j}-\frac{1}{2}.$$

$$\mathcal{D}$$

$$+\left(1-\sigma (b_j+⟨{\mathbf{w}}_j,{\mathbf{x}}_i⟩)\right)\log \left(\frac{1-\sigma (b_j+⟨{\mathbf{w}}_j,{\mathbf{x}}_i⟩)}{\frac{1}{n}{\sum }_{k=1}^{n}1-\sigma (b_j+⟨{\mathbf{w}}_j,{\mathbf{x}}_k⟩)}\right).$$

$${\mathcal{E}}_{\text{gap}}^D(q_{U|X},Q_{\hat{Y}|U},{\mathcal{S}}_n)=\left|{\mathcal{L}}^D(q_{U|X},Q_{\hat{Y}|U})-{\mathcal{L}}_{\text{emp}}^D(q_{U|X},Q_{\hat{Y}|U},{\mathcal{S}}_n)\right|$$

$${\mathcal{L}}^D(q_{U|X},Q_{\hat{Y}|U})$$

$${\mathcal{L}}_{\text{emp}}^D(q_{U|X},Q_{\hat{Y}|U},{\mathcal{S}}_n)$$

$$P_{XY}^D(x,y)=\sum _{k=1}^K\text{mathds}1\left\{x=x^{(k,y)}\right\}{\int }_{{\mathcal{K}}_k^{(y)}}{p}_{XY}(x^{\prime },y)d{x}^{\prime }.$$

$${\mathcal{E}}_{\text{gap}}(q_{U|X},Q_{\hat{Y}|U},{\mathcal{S}}_n)$$

$$+\left|\mathcal{L}(q_{U|X},Q_{\hat{Y}|U})-{\mathcal{L}}^D(q_{U|X},Q_{\hat{Y}|U})\right|$$

$$+\left|{\mathcal{L}}_{\text{emp}}^D(q_{U|X},Q_{\hat{Y}|U},{\mathcal{S}}_n)-{\mathcal{L}}_{\text{emp}}(q_{U|X},Q_{\hat{Y}|U},{\mathcal{S}}_n)\right|,$$

$$\left|\mathcal{L}(q_{U|X},Q_{\hat{Y}|U})-{\mathcal{L}}^D(q_{U|X},Q_{\hat{Y}|U})\right|$$

$$=\left|\sum _{k=1}^K\sum _{y=1}^{|\mathcal{Y}|}{P}_{XY}^D(x^{(k,y)},y)\left(\mathbb{E}\left[\ell (X,Y)|Y=y,X\in {\mathcal{K}}_k^{(Y)}\right]-\ell (x^{(k,y)},y)\right)\right|$$

$$\le ϵ(K)$$

$$\left|{\mathcal{L}}_{\text{emp}}^D(q_{U|X},Q_{\hat{Y}|U},{\mathcal{S}}_n)-{\mathcal{L}}_{\text{emp}}(q_{U|X},Q_{\hat{Y}|U},{\mathcal{S}}_n)\right|$$

$$=\frac{1}{n}\left|\sum _{k=1}^K\sum _{\begin{array}{c}i\in [1:n]\\ x_i\in {\mathcal{K}}_k\end{array}}\left[\ell (x^{(k,y_i)},y_i)-\ell (x_i,y_i)\right]\right|$$

$$\le ϵ(K).$$

$${\mathcal{E}}_{\text{gap}}^D(q_{U|X},Q_{\hat{Y}|U},{\mathcal{S}}_n)\le {\mathcal{E}}_{\text{gap}}^D(q_{U|X},Q_{Y|U}^D,{\mathcal{S}}_n)+d({\mathcal{S}}_n)$$

$$d({\mathcal{S}}_n)=\left|\frac{1}{n}\sum _{k=1}^K\sum _{\begin{array}{c}i\in [1:n]\\ x_i\in {\mathcal{K}}_k\end{array}}T(x^{(k,y_i)},y_i)-\mathcal{D}\left(Q_{Y|U}^D\Vert Q_{\hat{Y}|U}|q_U^D\right)\right|$$

$$\ell\big{(}q_{U|X}(\cdot|x),{Q}_{\hat{Y}|U}(y|\cdot)\big{)}=\ell\big{(}q_{U|X}(\cdot|x),{Q}^{D}_{Y|U}(y|\cdot)\big{)}+T(x,y)$$

$${\mathcal{E}}_{\text{gap}}^D$$

$$+\log \left(\frac{\text{Vol}\left(\mathcal{U}\right)}{P_Y(y_{\min })}\right)\sqrt{|\mathcal{Y}|}{\Vert {\mathbf{P}}_Y-{\hat{\mathbf{P}}}_Y\Vert }_2+\mathcal{O}\left(\Vert {\mathbf{P}}_Y-{\hat{\mathbf{P}}}_Y{\Vert }_2^2\right)$$

$$\displaystyle+\mathbb{E}_{P_{Y}}\Big{[}\int_{\mathcal{U}}\!\phi\left(\left\|\mathbf{P}_{X|Y}^{D}(\cdot|Y)\!-\!\mathbf{\hat{P}}_{X|Y}^{D}(\cdot|Y)\right\|_{2}\!\sqrt{\mathbb{V}\left(\mathbf{q}_{U|X}(u|\cdot)\right)}\right)du\Big{]}$$

$$\mathbb{V}(\mathbf{c}):={\Vert \mathbf{c}-\bar{c}\text{mathds}{1}_a\Vert }_2^2,$$

$${\mathcal{E}}_{\text{gap}}^D(q_{U|X},Q_{Y|U}^D,{\mathcal{S}}_n)$$

Full text

1 Introduction

Most theories of generalization for classification, both theoretical and empirical, assumes that models are trained and tested using data drawn from some fixed distribution. What is often needed in practice, however, is to learn a classifier which performs well on a target domain with a significantly different distribution of the training data, which may involve similar concepts that were observed previously by the learner but with some features changes (Bengio et al.,, 2019). In general, we would expect that the result of the learning phase to be able to generalize well to other distributions, with the added functionality of being able to monitoring the dynamics of the generalization due to external factors or non-stationarities of the involved distributions. In this paper, we investigate through the lens of information-theoretic principles how the testing gap between the empirical cross-entropy and its statistical expectation (computed from the target distribution) behaves when the true probability law of the data samples is potentially different at training and testing time.

According to classical statistical learning theory (Boucheron et al.,, 2005), models with many parameters tend to overfit by representing the training data too accurately, therefore diminishing their ability to generalize to unseen data Bishop, (2006). Interestingly enough, this phenomena does seem to be happening with Deep Neural Networks (DNNs) which even with many parameters and a modest number of training samples present good generalization properties as shown by  Neyshabur et al., (2017); Zhang et al., (2016).

Stochastic representations encompass the classical learning problem to include graphic models, even neuronal ones such as Variational Auto-Encoders (VAEs) (Kingma and Welling,, 2013) or Restricted Boltzmann Machines (RBMs) (Hinton,, 2012). With this models, mutual information between feature inputs and their representations becomes a relevant quantity. Empirical studies based on the Information Bottleneck (IB) method (Tishby et al.,, 1999) have shown that this mutual information may be related to the overfitting problem in DNNs. The IB method studies the tradeoff between accuracy and information complexity measured in terms of the mutual information. Statistical rates on the empirical estimates of the corresponding IB tradeoffs have been reported in Shamir et al., (2010). Schwartz-Ziv and Tishby, (2017) show, based on empirical evidence, that the DNN generalization capacity is induced by an implicit data compression of the feature inputs. However, these trade-offs are still controversial and the question remains open. As a matter of fact, recent works by (Amjad and Geiger,, 2018) and (Saxe et al.,, 2018) report difficulties in using the IB trade-off as an information-regularized objective for training. Indeed, it is known that estimating mutual information from high-dimensional data samples is a difficult and challenging problem for which, even state-of-the-art methods, might lead to misleading numerical results (Belghazi et al.,, 2018).

1.1 Our Contribution

We investigate the probability concentration of the testing gap between the empirical cross-entropy and its statistical expectation, measured with respect to the testing probability distribution which may be different of the training distribution. More specifically, there are two attributes that characterize our model: (a) we study a deviation on the target (testing) dataset once the training stage is finished, i.e., for a given soft-classifier; and (b) we consider randomized encoders which allows us not only to study deterministic feed-forward structures, but also stochastic models.

Theorem 1 provides the rigorous statement of the cross-entropy deviation bound that is the basis of this work. In contrast to standard learning concentration inequalities, this bound scales with $\log(n)/\sqrt{n}$ and $1/\sqrt{n}$ in a $n$-length dataset. Our bound depends on several factors: a mutual information term between the input and the representations generated from them using a reference selected encoder; a second term that measures the decoder efficiency; and two other magnitudes that measure how robust can be interpreted the problem, motivated by Xu and Mannor, (2012) among others. Despite the fact that our results may not lead to the tightest bounds, they are intended to reflect the importance of information-theoretic concepts in the problem of representation learning and the different trade-offs that can be established between information measures and quantities of interest in statistical learning.

An empirical investigation of the interplay between the cross-entropy deviation and the mutual information is provided on high-dimensional datasets of natural images with rotations and translations on the target domain. These simulations show the ability of mutual information to predict the behavior of the gap for the case of three well-known stochastic representation models: (a) the standard Variational Auto-Encoders (VAE) (Kingma and Welling,, 2013); (b) the log-normal encoder presented in the Information Dropout scheme in (Achille and Soatto,, 2016); and (c) the classical encoder based on Restricted Boltzmann Machines (RBMs) (Hinton,, 2012). These results validates the fact that mutual information is an important quantity that is strongly related to the generalization properties of learning algorithms, motivating the need of further studies in this respect.

The rest of the paper is organized as follows. In Section 2, we introduce a stochastic model which combines the randomized encoder concept like in Achille and Soatto, (2016) with the possibility of decomposing the classifier in a encoder and a decoder (Schwartz-Ziv and Tishby, (2017)). In Section 3 we present our general concentration inequality based on the above mentioned mutual information and other specific magnitudes, which will be discussed. While in Section 4 we show numerical evidence for some selected models, in Section 5 we provide concluding remarks. Major mathematical details are relegated to the appendices in the Appendix.

2 Representation and Statistical Learning

We are interested in the problem of pattern classification consisting in the prediction of the unknown class that match an observation. An observation or example is often a sample $x\in\mathcal{X}$ which have an associated label $y\in\mathcal{Y}$ (finite space). An $|\mathcal{Y}|$-ary classifier is defined by a (stochastic) decision rule $Q_{\hat{Y}|X}:\mathcal{X}\rightarrow\mathcal{P}(\mathcal{Y})$, where $\hat{Y}\in\mathcal{Y}$ denotes the random variable associated to the classifier output and $X$ is the random observed example. Typically, this classifier is trained with samples generated according to an unknown training distribution. It is also assumed that the testing examples and their corresponding labels are generated in an i.i.d. fashion according to $p_{XY}\coloneqq p_{X}P_{Y|X}$ (which could be possibly different from the above mentioned training distribution).111In writing this we allow us a little abuse of notation to simplify it since $p_{X}$ is a pdf. and $P_{Y|X}$ is pmf.

The problem of finding a good classifier can be divided into that of simultaneously finding an (possibly randomized) encoder $q_{U|X}:\mathcal{X}\rightarrow\mathcal{P}(\mathcal{U})$ that maps raw data to a higher-dimensional (feature) space $\mathcal{U}$ and a soft-decoder $Q_{\hat{Y}|U}:\mathcal{U}\rightarrow\mathcal{P}(\mathcal{Y})$ which maps the representation to a probability distribution on the label space $\mathcal{Y}$. These mappings induce an equivalent classifier:

[TABLE]

Remark** 1****.**

In the standard methodology with deep representations, we consider $L$ randomized encoders ($L$ layers) $\{q_{U_{l}|U_{l-1}}\}_{l=1}^{L}$ with $U_{0}\equiv X$. Although this appears at first to be more general, it can be casted formally using the one-layer case formulation induced by the marginal distribution that relates the input and the final $L$-th output layer. Therefore results on the one-layer formulation also apply to the $L$-th layer formulation and thus, we shall thus focus on the one-layer case without loss of generality.

This representation contains several cases of interest as the feed-forward neural net case as well as genuinely graphical model cases such as VAE or RBM. The computation of (1) requires marginalizing out $u\in\mathcal{U}$ which could be computationally prohibitive in practice. We use the cross-entropy as a loss-function:

[TABLE]

The learner’s goal is to select $(q_{U|X},Q_{\hat{Y}|U})$ by minimizing the expected risk under the training distribution: ${\mathcal{L}}(q_{U|X},Q_{\hat{Y}|U})\coloneqq\mathbb{E}_{p_{XY}}\left[\ell(X,Y)\right]$. This is done at training time using the empirical risk computed with i.i.d. samples from the training distribution. After training phase is over we would like to evaluate the performance of the obtained $(q_{U|X},Q_{\hat{Y}|U})$ based on a testing dataset: $\mathcal{S}_{n}\coloneqq\{(x_{1},y_{1})\cdots(x_{n},y_{n})\}$ which is independent from the training set (but not necessarily sampled from the same distribution). The testing risk is defined by

[TABLE]

Since the testing risk is evaluated on finite size samples, its evaluation may be sensitive to sampling noise error. The gap, to be defined next, is a measure of how an encoder-decoder pair could perform on unseen data (at training time) contained in the testing set $\mathcal{S}_{n}$.

Definition 1** (Error gap).**

Given a stochastic encoder $q_{U|X}:\mathcal{X}\rightarrow\mathcal{P}(\mathcal{U})$ and decoder $Q_{\hat{Y}|U}:\mathcal{U}\rightarrow\mathcal{P}(\mathcal{Y})$, the error gap is given by

[TABLE]

which quantifies the error associated to $(q_{U|X},Q_{\hat{Y}|U})$ when $\mathcal{L}_{\text{emp}}({q_{U|X},Q_{\hat{Y}|U}},{\mathcal{S}_{n}})$ is considered an estimate of $\mathcal{L}(q_{U|X},Q_{\hat{Y}|U})$ (calculated with the testing distribution).

Remark** 2****.**

The definition of this gap should not be confused with the typical one used for PAC-style bounds (Devroye et al.,, 1997). The latter is computed w.r.t. the training dataset while here we will study a deviation bound on the testing set, i.e., after the training stage has been accomplished. Our focus is reasonable for scenarios where the testing statistics evolves over time and may not match the training distribution.

3 Information Theoretic Bounds on the Gap

In this section, we first present our main result in Theorem 1, which is a bound on the gap (4) with probability at least $1-\delta$, as a function of a fixed randomized encoder-decoder pair $(q_{U|X},Q_{\hat{Y}|U})$. In particular, we show that the mutual information between the input raw data and its representation controls the gap with a scaling $\mathcal{O}\left(\frac{\log(n)}{\sqrt{n}}\right)$, which leads to a so-called informational deviation error bound. The information measures to be used in the work are Cover and Thomas, (2006): Kullback-Leibler (KL) divergence $\mathcal{D}({p}_{X}\|{q}_{X})\coloneqq\mathbb{E}_{p_{X}}\left[\log\frac{p_{X}(X)}{q_{X}(X)}\right]$; conditional KL divergence \mathcal{D}({p}_{Y|X}\|{q}_{Y|X}|{p}_{X})\coloneqq\mathbb{E}_{{p_{X}}}\left[\mathcal{D}\big{(}{p}_{Y|X}(\cdot|X)\|{q}_{Y|X}(\cdot|X)\big{)}\right] and mutual information $\mathcal{I}(p_{X};{p}_{Y|X})\coloneqq\mathcal{D}({p}_{Y|X}\|{{p}_{Y}}|p_{X})$. In this section, we will make use of the following assumptions:

Assumptions 1**.**

We assume $\mathcal{X}=\text{Supp}(p_{X})$ and $P_{Y}(y_{\min})\coloneqq\min_{y\in\mathcal{Y}}P_{Y}(y)>0$ without loss of generality because we can ignore the zero probability events. We assume that $\text{Vol}(\mathcal{U})<\infty$222The output encoder alphabet could be continuous. With this we are implying that it is bounded set. If $\mathcal{U}$ is discrete this assumption means that it is of finite cardinality. and that the selected encoder-decoder pair $(q_{U|X},Q_{\hat{Y}|U})$ is such that $\displaystyle Q_{\hat{Y}|U}(y_{\min}|u_{\min})\coloneqq\inf_{\begin{subarray}{c}y\in\mathcal{Y}\\ u\in\mathcal{U}\end{subarray}}Q_{\hat{Y}|U}(y|u)\geq\eta>0$.

The assumption of $Q_{\hat{Y}|U}(y_{\min}|u_{\min})>0$ is typically valid for the soft-max decoder, since in practice its parameters never diverge.

Theorem 1** (Information-theoretic bound).**

For every $\delta\in(0,1)$, with probability at least $1-\delta$ over the choice of $\mathcal{S}_{n}\sim p_{XY}$, the gap satisfies:

[TABLE]

$\forall\,(q_{U|X},Q_{\hat{Y}|U})$* that meets Assumptions 1, where $\mathcal{D}_{\text{HL}}$ is the Hellinger distance*

[TABLE]

constants are defined as $A_{\delta}\coloneqq\sqrt{2}B_{\delta}$, B_{\delta}\coloneqq\Big{(}1+\sqrt{\log\big{(}\frac{|\mathcal{Y}|+4}{\delta}\big{)}}\Big{)}, $C_{\delta}\coloneqq 2\text{Vol}\left(\mathcal{U}\right)e^{-1}$ $+B_{\delta}\sqrt{|\mathcal{Y}|}\log\left(\frac{\text{Vol}\left(\mathcal{U}\right)}{P_{Y}(y_{\min})}\right)$, $D_{\delta}=Q_{\hat{Y}|U}^{-1/4}(y_{\min}|u_{\min})\sqrt{8\frac{|\mathcal{Y}|+4}{\delta}}$; and

[TABLE]

where $\left(\{\mathcal{K}_{k}^{(y)}\}_{k=1}^{K},\{x^{(k,y)}\}_{k=1}^{K}\right)_{y\in\mathcal{Y}}$ are $|\mathcal{Y}|$ partitions of $\mathcal{X}$ and their respective centroids. They are clearly functions of the natural number $K$ and such that for each $y\in\mathcal{Y}$: $\bigcup_{k=1}^{K}\mathcal{K}_{k}^{(y)}=\mathcal{X},\;\mathcal{K}_{i}^{(y)}\cap\mathcal{K}_{j}^{(y)}=\emptyset\;\forall 1\leq i<j\leq K,\;\text{Vol}(\mathcal{K}_{k}^{(y)})>0\;\forall 1\leq k\leq K$; and

[TABLE]

[TABLE]

are distributions functions induced by the quantization of the testing distribution $p_{XY}(x,y)$ by the above mentioned partitions.

The proof is relegated to the Appendix. This bound has some important terms which are worth analyzing:

• •

Encoder and decoder $(q_{U|X},Q_{\hat{Y}|U})$: the encoder and the decoder in Theorem 1 are considered to be given. However, without loss of generality they can correspond to the learned encoder and decoder during training time. In this case, the bound in (5) should be further averaged with respect to the randomness of the training samples –according to the training distribution– and the training algorithm itself.

• •

$\mathcal{I}(p_{X};q_{U|X})$: Mutual information between raw data $X$ and its randomized representation $U$ is a regularization term used to reduce the overfitting, that some authors understand as “measure of information complexity” (Achille and Soatto,, 2016; Alemi et al.,, 2016; Vera et al.,, 2018). Theo. 1 is a first step in order to explain how and why this effect happens. This term presents a scaling rate of $n^{-1/2}\log(n)$ and it is the most important term of our deviation bound. In Section 4 we present a empirical analysis in order to study its importance.

• •

$\mathcal{D}_{\text{HL}}\left(Q^{D}_{Y|U}\|Q_{\hat{Y}|U}\right)$: Hellinger distance could be seen as a measure of the decoder efficiency in comparison with the decoder $Q^{D}_{Y|U}$, which is induced by the randomized encoder $q_{U|X}$ and the quantized testing distribution. When $Q_{\hat{Y}|U}=Q^{D}_{Y|U}$ this term is zero, suggesting that this selection minimizes the error gap. In this way, we are interested in decoders with sufficient freedom degrees as the soft-max decoder.

• •

$\epsilon(K)$ and $r(K)$: Motivated by robust quantization (Xu and Mannor,, 2012), these functions define, for each $y\in\mathcal{Y}$, an artificial discretization of $\mathcal{X}$ space into cells (partition element). This discretization allow us to introduce some information-theoretic techniques and results for discrete alphabets during the proof of our result. While $\epsilon(K)$ is associated with the robustness of the loss function over the partition element, $r(K)$ is the minimum probability of falling into a cell. There is a tradeoff between these: while $\epsilon(K)$ is a decreasing function (when the number of cells is increased, they may be smaller), $r(K)$ is a increasing one (smaller cells enclose less probability).

• •

$Q_{\hat{Y}|U}(y_{\min}|u_{\min})$: The maximum of the loss function value could be a poor choice to generate a deviation bound with dependence on the decoder. In our case, a more sensible approach is followed by using the Hellinger distance between $Q_{\hat{Y}|U}$ and $Q^{D}_{Y|U}$. It can be seen that this decoder dependent term will not be relevant in two scenarios: when decoder selection $Q_{\hat{Y}|U}$ is close to $Q^{D}_{Y|U}$ (in a Hellinger distance sense) and when $Q_{\hat{Y}|U}(y_{\min}|u_{\min})$ is small enough (through $D_{\delta}$). The consideration of these two possibilities and the $1/\sqrt{n}$ scaling of this term could justify to disregarding it when the number of samples is large enough.

• •

$\text{Vol}(\mathcal{U})$: Note that if ReLU activations are implemented (whose volume is limited for bounded entries), $\text{Vol}(\mathcal{U})$ is expected to be larger than for the case of sigmoid activations. As a consequence, the mutual information will have a major influence in the generalization with saturated activations. This observation matches with Saxe et al., (2018) analysis.

4 Experimental Results

In this section, we experimentally check our bound in some stochastic models used in practice. We will show that the mutual information is representative of the gap behavior, i.e. we compare, after a training stage, a $\mathcal{E}_{\textrm{gap}}(q_{U|X},Q_{\hat{Y}|U},\mathcal{S}_{n})$ quantile (the magnitude to be bounded) with $A\cdot\sqrt{\mathcal{I}(p_{X};q_{U|X})}+C$, where $A$ and $C$ are universal constants representative of the corresponding ones in Theorem 1. The magnitudes are compared for several rules $Q$ indexed by a Lagrange multiplier defined in each experiment. That is, we first select an encoder $q_{U|X}$ and a decoder $Q_{\hat{Y}|U}$ based on a Lagrange multiplier $\lambda$ in the cost function and the training set, and then we evaluate the risk deviation based on independent features to those of used during the training, i.e., according to the testing dataset $\mathcal{S}_{n}$. We focus on the qualitative behavior, so we plot with different axes for the quantile gap and mutual information in order to get rid of the mentioned constants.

We will show that the mutual information is representative of the behavior of said gap, even when the distribution with which the test samples are generated does not match with the training law. As our main goal is not to present a new classification methodology, with competitive results with state-of-the-art methods, we restrict ourself to use small databases, as motivated in Neyshabur et al., (2017) work. We sample a random subset of MNIST (standard dataset of handwritten digits). The size of the training set is $5K$ and the algorithms will be tested with standard MNIST dataset and with a disturbed version with translations and rotations. Random translations are drawn from an uniform distribution between -5 and 5 (quantized) for each axis and random rotations are drawn from an uniform distribution $(-\frac{\pi}{4},\frac{\pi}{4})$ for the angle, as can be seen in Fig. 1. Experiments with CIFAR-10 dataset (natural images) can be seen in Appendix.

In most applications, alphabets are such that $\mathcal{X}\subset\mathbb{R}^{d}$, $\mathcal{U}\subset\mathbb{R}^{m}$, where $d$ is the number of input units and $m$ is the number of hidden units of the auxiliary variable, so we refer to the vector random variables $\mathbf{X},\mathbf{U}$ respectively. We approximate $\mathcal{L}(q_{U|X},Q_{\hat{Y}|U})$ with a $5K$ dataset and $\mathcal{L}_{\text{emp}}(q_{U|X},Q_{\hat{Y}|U},{\mathcal{S}_{n}})$ with different independent mini-testing datasets of $100$ samples, i.e., using the rest of the features. The $\mathcal{E}_{\textrm{gap}}(q_{U|X},Q_{\hat{Y}|U},\mathcal{S}_{n})$ $0.95$-quantile ($\delta=0.05$) is computed based on the different values of the testing risk: $\mathcal{L}_{\text{emp}}(q_{U|X},Q_{\hat{Y}|U},{\mathcal{S}_{n}})$. Finally, there exists the difficulty of implementing a mutual information estimator. To this end, we make use of the variational bound (Cover and Thomas,, 2006) to upper bound the mutual information by \mathcal{I}\left(p_{\mathbf{X}};q_{\mathbf{U}|\mathbf{X}}\right)\leq\mathcal{D}\left(q_{\mathbf{U}|\mathbf{X}}\big{\|}\tilde{q}_{\mathbf{U}}\big{|}p_{\mathbf{X}}\right), where $\tilde{q}_{\mathbf{U}}$ is an auxiliary prior pdf (Kingma and Welling,, 2013; Achille and Soatto,, 2016). Consider a distribution $\tilde{q}_{U}$ a product distribution $\tilde{q}_{\mathbf{U}}(\mathbf{u})=\prod_{j=1}^{m}\tilde{q}_{U_{j}}(u_{j})$. It is straightforward to check that

[TABLE]

We will implement a parametric estimation of the KL divergence \mathcal{D}\left(q_{U_{j}|\mathbf{X}}(\cdot|\mathbf{X})\big{\|}\tilde{q}_{U_{j}}\big{|}\hat{P}_{\mathbf{X}}\right), where $\hat{P}_{\mathbf{X}}$ is the empirical pmf (KL estimation is an average over the sample), for each of the following architectures: normal encoder / normal prior Kingma and Welling, (2013), log-normal encoder / log-normal prior Achille and Soatto, (2016) and RBM encoder / $\frac{1}{n}\sum_{i=1}^{n}q_{U_{j}|\mathbf{X}}(u_{j}|\mathbf{x}_{i})$ prior Hinton, (2012). We will refer to each of the examples by their encoders: Normal, log-Normal and RBM, respectively. Note that the mutual information does not depend on the decoder, so we use always a soft-max output layer for simplicity. Experimental details can be seen in Appendix.

4.1 Normal Encoder: Variational Classifier

Gaussian Variational Auto-Encoders (VAEs) introduce a normal encoder $U_{j}|_{\mathbf{X}=\mathbf{x}}\sim\mathcal{N}(\mu_{j}(\mathbf{x}),\sigma^{2}_{j}(\mathbf{x}))$ $j=[1:m]$, where $\mu_{j}(\mathbf{x})$ and $\log\sigma_{j}^{2}(\mathbf{x})$ are constructed via DNNs (vectorized), a standard normal prior $\tilde{U}_{j}\sim\mathcal{N}(0,1)$ and the decoder input is generated by simple sampling using the reparameterization trick Kingma and Welling, (2013). In this case each KL divergence in (10) can be estimated as follows:

[TABLE]

Fig. 2(a) and 2(b) show $0.95$-quantile of $\mathcal{E}_{\textrm{gap}}(q_{U|X},Q_{\hat{Y}|U},\mathcal{S}_{n})$ and the mutual information variational bound (10) as a function of the training Lagrange multiplier $\lambda$, testing with the training and disturbed MNIST dataset respectively. There is an decreasing tendency indicating the vanishing of possible overfitting when regularization is incremented. Complex tasks, such as testing with images sampled with another distribution Fig. 2(b), generate an error gap behavior closer to the mutual information.

4.2 Log-Normal Encoder: Information Dropout

Information dropout propose log-normal encoders $U_{j}=f_{j}(\mathbf{X})e^{\alpha_{j}(\mathbf{X})Z}$ $j=[1:m]$ where $Z\sim\mathcal{N}(0,1)$, where $f_{j}(\mathbf{x})$ and $\alpha_{j}^{2}(\mathbf{x})$ are constructed via DNNs (vectorized) and the decoder input is generated by simple sampling using the reparameterization trick. In Achille and Soatto, (2016), authors recommend to use a log-normal prior $\tilde{U}_{j}\sim\log\mathcal{N}(\mu_{j},\sigma_{j}^{2})$ when $\mathbf{f}(\mathbf{x})=[f_{1}(\mathbf{x}),\cdots f_{m}(\mathbf{x})]$ is a DNN with soft-plus activation, where $\mu_{j}$ and $\sigma_{j}$ are variables to train.

As $U_{j}|_{\mathbf{X}=\mathbf{x}}\sim\log\mathcal{N}(\log f_{j}(\mathbf{x}),\alpha^{2}_{j}(\mathbf{x}))$ and the KL divergence is invariant under reparametrizations, the divergence between two log-normal distributions is equal to the divergence between the corresponding normal distributions. Therefore, using the formula for the KL divergence of normal random variables Cover and Thomas, (2006), we obtain

[TABLE]

Fig. 3(a) and 3(b) show $0.95$-quantile of $\mathcal{E}_{\textrm{gap}}(q_{U|X},Q_{\hat{Y}|U},\mathcal{S}_{n})$ and the mutual information variational bound (10) as a function of the training Lagrange multiplier $\lambda$, testing with the training and disturbed MNIST dataset respectively. Again, there is an decreasing tendency indicating the vanishing of possible overfitting when regularization is incremented. In this case the behaviors are pretty close because the prior distribution is also trained.

4.3 RBM Encoder: Classification using Restricted Boltzmann Machines

Consider the standard models for the RBM studied in Hinton, (2012); Srivastava et al., (2014). For every $j\in[1:m]$, $U_{j}$ given $\mathbf{X}=\mathbf{x}$ is distributed as a Bernoulli RV with parameter $\sigma(b_{j}+\mathbf{w}_{j}^{T}\mathbf{x})$ (sigmoid activation). Selecting the product distribution $\tilde{q}_{U_{j}}(u_{j})=\frac{1}{n}\sum_{i=1}^{n}q_{U_{j}|\mathbf{X}}(u_{j}|\mathbf{x}_{i})$, we obtain

[TABLE]

Figures 4(a) and 4(b) show $0.95$-quantile of $\mathcal{E}_{\textrm{gap}}(q_{U|X},Q_{\hat{Y}|U},\mathcal{S}_{n})$ and the mutual information variational bound (10) as a function of the Lagrange multiplier $\lambda$, testing with the training and disturbed MNIST dataset respectively. Again, the behaviors are similar and decreasing, both when testing with samples generated with the training distribution and with samples generated with another one.

5 Summary and Concluding Remarks

We presented a theoretical investigation of a typical classification task in which we have training data from a source domain, but we wish the testing gap between the empirical cross-entropy and its statistical expectation (measured with respect to a possible different testing probability law) to be as small as possible. Our main result (Theorem 1) is that the testing gap can be bounded with high probability by the mutual information between the input testing samples and the corresponding representations, the Hellinger distance which measures the decoder efficiency and other less relevant constants. Empirical study of this metric suggests that the mutual information may be a good measure to capture the dynamic of the gap with respect to important training parameters. We finally presented a simple experimental setup which shows that there is a strong correlation between the gap represented and the mutual information between the raw inputs and their representations. Some further work will be needed to provide strong support to these numerical results in presence of other sources of non-stationarities between training and testing datasets.

Acknowledgment

The work of Prof. Pablo Piantanida was supported by the European Commission’s Marie Sklodowska-Curie Actions (MSCA), through the Marie Sklodowska-Curie IF (H2020-MSCAIF-2017-EF-797805). The work of Matias Vera was supported by Peruilh PhD Scholarship from Facultad de Ingeniería, Universidad de Buenos Aires.

Notation and conventions

Let $\mathcal{P}(\mathcal{X})$ denote the set of all probability measures $\mathbb{P}$ over the set $\mathcal{X}$. The probability mass (pmf) function or probability density function (pdf) in the case of continuous random variables of $X$ is denoted interchangeably by $P_{X}$ or $p_{X}$. $\text{Supp}(p_{X})$ denotes the support of the distribution, i.e. the closure of $\{x\in\mathcal{X}:p_{X}(x)>0\}$. $\text{Vol}\left(\mathcal{X}\right)=\int_{\mathcal{X}}dx\;$ if $\;\mathcal{X}$ is continuous or $\text{Vol}\left(\mathcal{X}\right)=|\mathcal{X}|\;$ if $\;\mathcal{X}$ is discrete. $\|\cdot\|_{2}$ denotes the usual Euclidean norm of a vector and $\langle\cdot,\cdot\rangle$ the canonical inner product. We use $\mathbb{E}_{p}[\cdot]$ and $\text{Var}_{p}(\cdot)$ to denote the mathematical expectation and variance respectively, measured with respect to $p$ . The information measures to be used in the work are Cover and Thomas, (2006): the entropy $\mathcal{H}(P_{X})\coloneqq\mathbb{E}_{P_{X}}\left[-\log{{P}_{X}}(X)\right]$; the conditional entropy $\mathcal{H}(P_{Y|X}|P_{X})\coloneqq\mathbb{E}_{P_{X}P_{Y|X}}\left[-\log{P}_{Y|X}(Y|X)\right]$; the relative entropy: $\mathcal{D}({P}_{X}\|{Q}_{X})\coloneqq\mathbb{E}_{P_{X}}\left[\log\frac{P_{X}(X)}{Q_{X}(X)}\right]$; the conditional relative entropy: \mathcal{D}({P}_{Y|X}\|{Q}_{Y|X}|{P}_{X})\coloneqq\mathbb{E}_{{P_{X}}}\left[\mathcal{D}\big{(}{P}_{Y|X}(\cdot|X)\|{Q}_{Y|X}(\cdot|X)\big{)}\right] and the mutual information: $\mathcal{I}(P_{X};{P}_{Y|X})\coloneqq\mathcal{D}({P}_{Y|X}\|{{P}_{Y}}|P_{X})$. We talk about testing dataset $\mathcal{S}_{n}$ of $n$ samples. We study a deviation bound, after training (over $\mathcal{S}_{n}$), so we assume that the training set is given implicitly.

Appendix A Proof of Theorem 1

In this Appendix we will prove the main result of this work. We will use some well-known results, listed in C. We are looking for a relationship between error gap and mutual information, which is a information-theoretic measure. Information theory in general and mutual information in particular has several known results, a large percentage of them being for discrete spaces. For example, Shamir et al., (2010) bound the deviation over the mutual information between labels $Y$ and hidden representations $U$, through the mutual information between hidden representations and inputs $X$ with discrete alphabets. However, in most of learning problems it is more appropriate to consider a continuous alphabet $\mathcal{X}$. In order to use those discrete results, our first step is to analyze the error introduced due to a reasonable discretization. This approach has points in common with Xu and Mannor, (2012) robust-algorithms theory.

Lemma 1**.**

Let $|\mathcal{Y}|$ different partitions of $\mathcal{X}$ and their respective centroids for each $y\in\mathcal{Y}$ $\left(\{\mathcal{K}_{k}^{(y)}\}_{k=1}^{K},\{x^{(k,y)}\}_{k=1}^{K}\right)$ function of the natural number $K$, where partitions meet for each $y\in\mathcal{Y}$: $\bigcup_{k=1}^{K}\mathcal{K}_{k}^{(y)}=\mathcal{X},\;\mathcal{K}_{i}^{(y)}\cap\mathcal{K}_{j}^{(y)}=\emptyset\;\forall 1\leq i<j\leq K,\;\int_{\mathcal{K}_{k}^{(y)}}dx>0\;\forall 1\leq k\leq K$, the error gap (4) can be bounded as $\mathcal{E}_{\textrm{gap}}(q_{U|X},Q_{\hat{Y}|U},\mathcal{S}_{n})\leq 2\epsilon(K)+\mathcal{E}_{\textrm{gap}}^{D}(q_{U|X},Q_{\hat{Y}|U},\mathcal{S}_{n})$ almost surely, where $\epsilon(K)$ was defined in (7) and $\mathcal{E}_{\textrm{gap}}^{D}(q_{U|X},Q_{\hat{Y}|U},\mathcal{S}_{n})$ is defined as

[TABLE]

where

[TABLE]

and $P^{D}_{XY}:\;\{x^{(k,y)}:\;k\in[1:K],y\in\mathcal{Y}\}\times\mathcal{Y}\rightarrow[0,1]$ is

[TABLE]

On the one hand, $P^{D}_{XY}$ is a pmf such that $P_{Y}$ (true value) is its marginal. On the other hand the other marginal $P_{X}^{D}(x)=\sum_{y\in\mathcal{Y}}P^{D}_{XY}(x,y)$ has the elements of the set $\mathcal{A}=\{x^{(k,y)}:\;1\leq k\leq K,\;y\in\mathcal{Y}\}$ as atoms.

Proof.

Triangle inequality allow to relate error gaps as,

[TABLE]

where the last term is the discrete error gap $\mathcal{E}_{\textrm{gap}}^{D}(q_{U|X},Q_{\hat{Y}|U},\mathcal{S}_{n})$. The other terms in (19) can be bounded using the fact that $P_{X}^{D}(x^{(k,y)})=\mathbb{P}\left(X\in\mathcal{K}_{k}^{(Y)},Y=y\right)$ and definition of $\epsilon(K)$ (7):

[TABLE]

∎

The second step in our proof is to measure the decoupling between the encoder and decoder, i.e. what is the error when considering $Q^{D}_{Y|U}$ (9) as decoder. The following lemma separates the decoder term.

Lemma 2**.**

Discrete error gap can be bounded as

[TABLE]

almost surely, where

[TABLE]

and $T(x,y)\coloneqq\mathbb{E}_{q_{U|X}}\left[\left.\log\left(\frac{Q^{D}_{Y|U}(y|U)}{Q_{\hat{Y}|U}(y|U)}\right)\right|X=x\right]$.

Note that $\mathbb{E}_{P^{D}_{XY}}\left[T(X,Y)\right]=\mathcal{D}\left(Q^{D}_{Y|U}\|Q_{\hat{Y}|U}|q^{D}_{U}\right)$, so $d(\mathcal{S}_{n})$ is a deviation of $T(X,Y)$.

Proof.

It is immediate to see that:

[TABLE]

So, taking expectation (as (16) and (17)) and using triangle inequality, we can prove the lemma. ∎

The third step of our proof is to bound the new gap $\mathcal{E}_{\textrm{gap}}^{D}(q_{U|X},Q^{D}_{Y|U},\mathcal{S}_{n})$. For that, we use empirical distributions $\hat{P}^{D}_{XY},\hat{P}^{D}_{X},\hat{P}_{Y},\hat{P}^{D}_{X|Y}$ as the occurrence rate of $\mathcal{S}_{n}$; e.g. $\hat{P}^{D}_{XY}(k,y)=\frac{|\left\{(x_{i},y_{i})\in\mathcal{S}_{n}:\;y_{i}=y,\;x_{i}\in\mathcal{K}_{k}^{(y)}\right\}|}{n}$. Also, we define the empirical distributions generated from the encoder $\hat{q}^{D}_{U},\hat{Q}^{D}_{Y|U},\hat{q}^{D}_{U|Y}$ generated from the encoder; e.g. $q_{U}^{D}(u)=\sum_{x\in\mathcal{A}}q_{U|X}(u|x)\hat{P}^{D}_{X}(x)$.

Remark** 3****.**

The marginal $P_{Y}$ and its empirical pmf $\hat{P}_{Y}$ has not got superscript $D$ because it matches with the true pmf (without quantization).

Lemma 3**.**

The gap $\mathcal{E}_{\textrm{gap}}^{D}(q_{U|X},Q^{D}_{Y|U},\mathcal{S}_{n})$ can be bounded as,

[TABLE]

almost surely, where $\phi(\cdot)$ is defined in (64) and

[TABLE]

with $\mathbf{c}\in\mathbb{R}^{a}$, $a\in\mathbb{N}$, $\bar{c}=\frac{1}{a}\sum_{i=1}^{a}c_{i}$, and $\mathds{1}_{a}$ is the vector of ones of length $a$.

Notation $\mathbf{P}_{Y}$ means the pmf $P_{Y}$ as a vector $\mathbf{P}_{Y}=[P_{Y}(1),\cdots,P_{Y}(|\mathcal{Y}|)]$, so we can apply it vector norms $\|\cdot\|_{2}$ and $\mathbb{V}(\cdot)$ operator. This operator measures the dispersion of the components of $\bar{a}$ around the mean, where $\mathbb{V}(\mathbf{c})\leq\left\|\mathbf{c}-b\mathds{1}_{a}\right\|_{2}^{2}\ ,\ \forall b\in\mathbb{R}$.

Proof.

Adding and subtracting $\hat{P}_{XY}^{D}(x^{(k,y)},y)\mathbb{E}_{q_{U|X}}\left[\left.\log\left(\frac{1}{\hat{Q}^{D}_{Y|U}(y|U)}\right)\right|X=x^{(k,y)}\right]$ we can prove via triangle inequality that

[TABLE]

We can bound the second term in (30) using the inequality:

[TABLE]

The first term of (30) can be bounded as:

[TABLE]

where $\mathcal{H}_{d}$ is the differential entropy when $U$ is continuous and the classical entropy when $U$ is discrete. The terms $\left|\mathcal{H}_{d}(q^{D}_{U})\!-\!\mathcal{H}_{d}(\hat{q}^{D}_{U})\right|$ and $\left|\mathcal{H}_{d}(q^{D}_{U|Y}|P_{Y})-\mathcal{H}_{d}(\hat{q}^{D}_{U|Y}|\hat{P}_{Y})\right|$ can be bounded by Lemmas 9 and 10 respectively. Finally, it is clear that $P_{Y}\mapsto\mathcal{H}(P_{Y})$ is differentiable and a first order Taylor expansion yields:

[TABLE]

where $\frac{\partial\mathcal{H}(P_{Y})}{\partial P_{Y}(y)}=-\log P_{Y}(y)-1$ for each $y\in\mathcal{Y}$. Then, applying Cauchy-Schwartz inequality the lemma was proved:

[TABLE]

∎

The combination of lemmas 1, 2 and 3 allow to bound the error gap with probability one. The fourth step is to use a concentration inequality over the terms: \mathcal{D}\big{(}\hat{P}_{XY}^{D}\|P_{XY}^{D}\big{)}, $\|\mathbf{P}_{X}^{D}-\mathbf{\hat{P}}_{X}^{D}\|_{2}$, $\|\mathbf{P}_{Y}-\mathbf{\hat{P}}_{Y}\|_{2}$, $\|\mathbf{P}_{X|Y}^{D}(\cdot|y)-\mathbf{\hat{P}}_{X|Y}^{D}(\cdot|y)\|_{2}$ for $y\in\mathcal{Y}$ and $d(\mathcal{S}_{n})$ simultaneously. Lemma 7 guarantees that the bounds hold simultaneously over all these $|\mathcal{Y}|+4$ quantities, by replacing $\delta$ with $\delta/(|\mathcal{Y}|+4)$. With probability at least $1-\delta$, we apply Lemmas 6, 8 and Chebyshev inequality Devroye et al., (1997):

[TABLE]

In order to bound the last variance we enunciate the following lemma.

Lemma 4**.**

Variance of $T$ random variable can be bounded as

[TABLE]

where $h$ is the Hellinger distance (6).

Proof.

A similar result can be founded Ghosal et al., (2000). Variance can be bounded as

[TABLE]

Note that $\frac{1}{Q_{\hat{Y}|U}(y_{\min}|u_{\min})}\geq\sup\left\{\frac{Q^{D}_{Y|U}(y|u)}{Q_{\hat{Y}|U}(y|u)},1\right\}$. For every $c\leq 0$ and $x\geq c$ we have the inequality $x^{2}\leq 4e^{-c/2}\left(e^{x/2}-1\right)^{2}$. So call $c=\log Q_{\hat{Y}|U}(y_{\min}|u_{\min})$ and $x=\log\frac{Q_{\hat{Y}|U}(y|u)}{Q^{D}_{Y|U}(y|u)}$, the inequality can be written as:

[TABLE]

Taking expectation term by term over $q_{U}^{D}Q^{D}_{Y|U}$ the proof is over. ∎

We generate the following bound for the error gap using concentration inequalities (37), (38) and (39).

Lemma 5**.**

For every $\delta\in(0,1)$, with probability at least $1-\delta$ over the choice of $\mathcal{S}_{n}\sim p_{XY}$, the gap satisfies:

[TABLE]

$\forall\,(q_{U|X},Q_{\hat{Y}|U})$* that meets Assumptions 1.*

Proof.

We bound the error gap using lemmas 1, 2, 3, 4 and 11, with probability at least $1-\delta$:

[TABLE]

We relate the mutual information $\mathcal{I}(P_{X}^{D};q_{U|X})$ with $\int_{\mathcal{U}}\sqrt{\mathbb{V}\left(\mathbf{q}_{U|X}(u|\cdot)\right)}du$. Its proof follows from an application of Pinsker’s inequality (Cover and Thomas,, 2006, Lemma 11.6.1) $\|\mathbf{P}_{1}-\mathbf{P}_{2}\|_{1}^{2}\leq 2\mathcal{D}(P_{1}\|P_{2})$ and the fact that $\mathbb{V}(\mathbf{c})\leq\left\|\mathbf{c}-b\mathds{1}_{a}\right\|_{2}^{2}\ ,\ \forall b\in\mathbb{R}$:

[TABLE]

where $Q^{D}_{X|U}(k|u)=\frac{q_{U|X}(u|x^{(k)})P_{X}^{D}(k)}{q^{D}_{U}(u)}$. So, using Jensen inequality

[TABLE]

for all $K\in\mathbb{N}$. Finally the lemma is proved taking the infimum over $K$. ∎

Finally, using the identity $\mathcal{I}(p_{X};q_{U|X})=\mathcal{I}(p_{XY};q_{U|X})$ and the fact that the $k$ where a sample $x$ belongs is a deterministic function of $x$ and $y$, we can use the Data Processing Inequality Cover and Thomas, (2006) $\mathcal{I}(P_{X}^{D};q_{U|X})\leq\mathcal{I}(p_{XY};q_{U|X})=\mathcal{I}(p_{X};q_{U|X})$. With this result the lemma 5 becomes in our Theorem 1.

Remark** 4****.**

It is worth mentioning the differences between our result and that presented in Shamir et al., (2010). While we work with the cross-entropy gap making appear several terms in lemma 3, their only bounded the mutual information gap:

[TABLE]

For this reason we had to use more concentration inequalities including (37) and Chebyshev, while their work only uses Lemma 8. In addition, our final mutual information is expressed in terms of continuous pdf, while theirs is a function of the discrete pmf. Finally some constants were subtly reduced and we extend the result for continuous representations of $U$.

Appendix B Experimental Details and Other Results

In this section we mention the details of the simulations in Section 4, and present new simulations using the CIFAR-10 dataset (Krizhevsky,, 2009). We sample two different random subsets of: MNIST (standard dataset of handwritten digits) and CIFAR-10 (natural images). The size of the training set is $5K$ for both datasets. It is important to emphasize the main difference between the datasets: while MNIST propose a task that is simpler, CIFAR-10 is a more difficult one, especially with a small numbers of samples and without convolutional DNNs. From this observation, we would expect that in the latter case, the classifier will not be trained enough to have an close to optimal performance. We approximate $\mathcal{L}(q_{U|X},Q_{\hat{Y}|U})$ with a $5K$ dataset and $\mathcal{L}_{\text{emp}}(q_{U|X},Q_{\hat{Y}|U},{\mathcal{S}_{n}})$ with different independent mini-testing datasets of $100$ samples, i.e., using the rest of the features. The $\mathcal{E}_{\textrm{gap}}(q_{U|X},Q_{\hat{Y}|U},\mathcal{S}_{n})$ $0.95$-quantile ($\delta=0.05$) is computed based on the different values of the testing risk. The values reported in each simulation are the average of three simulations independent, choosing at random, in each case, different sets. Below are the architectures used in each setup:

• •

Normal encoder: The DNNs used was a feed-forward layer of $512$ hidden units with ReLU activation followed by another linear for each parameter ($\mu$ and $\log\sigma^{2}$) with $256$ hidden units. I.e., each parameter, $\mu$ and $\log\sigma^{2}$, are a two-layers network where the first one is common to both. We choose a learning rate in $0.001$, a batch-size of $100$ and we train during $200$ epochs. The cost function considered during the training phase was of the form,

[TABLE]

where $\lambda$ is the regulation Lagrange multiplier and $\mathcal{D}_{l}$ is the $l$-training dataset.

• •

LogNormal encoder: The DNNs used for $\mathbf{f}(\mathbf{x})$ was a feed-forward structure with two layers of $256$ hidden units with a softplus activation and for $\boldsymbol{\alpha}(\mathbf{x})$ a feed-forward layer of $256$ hidden units with a sigmoid activation multiplied by $0.7$, so that the maximum variance of the log-normal error distribution will be approximately $1$ Achille and Soatto, (2016). We choose a learning rate in $0.001$, a batch-size of $100$ and we train during $200$ epochs. The cost function trained was the same that the one in (56).

• •

RBM encoder: Eq. (14) is difficult to use as a regularizer even using training with the contrastive divergence learning procedure Hinton, (2002). Instead, we rely on the usual RBM regularization: weight-decay. This is a traditional way to improve the generalization capacity. We explore the effect of the Lagrange multiplier $\lambda$, so called weight-cost, over both the gap and the mutual information. This meta-parameter controls the gradient weight decay, i.e., the cost function can be written as:

[TABLE]

where $CD_{\text{RBM}}$ is the classical unsupervised RBM cost function trained with the contrastive divergence learning procedure Hinton, (2012), and $\mathbf{W}$ is the matrix that has $\mathbf{w}_{j}$, with $j\in[1:m]$, as columns. In order to compute the gap we add to the output of the last RBM layer a soft-max regression decoder trained during $500$ epochs separately. Several authors have combined RBMs with soft-max regression, among them Hinton et al., (2006), Srivastava et al., (2013) and Chopra and Yadav, (2018).

Following suggestions from Hinton, (2012), we study the Lagrange multiplier when $\lambda\geq 0.00001$. We choose learning rates in $0.1$, a batch-size of $100$, $256$ hidden units and we train during $200$ epochs. We start with a momentum of $0.5$ and change to $0.9$ after $5$ epochs.

The following subsections present new experiment using CIFAR-10 dataset: Comparison of the behavior of the error gap and mutual information, and an example about the tradeoff between $\epsilon(K)$ and $r(K)$.

B.1 Comparison between the behavior of the error gap and mutual information

In this section we repeat simulation of Section 4 for CIFAR-10 dataset, testing with images without disturbing. Figures 5(a), 5(b) and 5(c) show $0.95$-quantile of $\mathcal{E}_{\textrm{gap}}(q_{U|X},Q_{\hat{Y}|U},\mathcal{S}_{n})$ and the mutual information variational bound (10) as a function of the Lagrange multiplier $\lambda$ for Normal encoder, logNormal encoder and RBM encoder respectively. In these cases, mutual information and error gap have a behavior closer than MNIST.

B.2 Discretization Tradeoff with CIFAR-10 dataset

We use the setup presented in Section 4.1 to implement numerically this tradeoff between $\epsilon(K)$ and $r(K)$ through K-Means algorithm. For every $K$, we iterate between:

• •

The samples coloring: Let the loss centroids $\{\ell(x^{(k)},y)\}_{k=1}^{K}$ for each $y\in\mathcal{Y}$, we assign $x_{i}\in\mathcal{K}_{k_{i}}$, where $k_{i}$ is computed as

[TABLE]

• •

Find centroids: We compute for each333We compute the loss centroid inside the true centroid $x^{(k)}$. $k$ and each $y\in\mathcal{Y}$

[TABLE]

After that, we estimate $\epsilon(K)$ and $r(K)$ as:

[TABLE]

using a $5K$ dataset of CIFAR-10. This algorithm focus on minimize $\epsilon(K)$ inside the tradeoff, and this function is really sensible to the samples (it only depends of the maximum). Fig. 6 shows a noisy tradeoff, which achieves its minimum at $K=16$.

Appendix C Auxiliary Results

In this appendix some auxiliary facts, which are used in proof of the main results, are listed without proof.

Lemma 6** (Theorem 11.2.1 (Cover and Thomas,, 2006)).**

Let ${P}\in\mathcal{P}(\mathcal{X})$ be a discrete probability distribution and let $\hat{P}$ be its empirical estimation over a $n$-data set $\mathcal{S}_{n}$. Then,

[TABLE]

with probability at least $1-\delta$ over the choice of $\mathcal{S}_{n}$.

Lemma 7** (Union bound application).**

*Let $\{\mathcal{A}_{k}\}_{k=1}^{m}$ events such that $\Pr(\mathcal{A}_{k})\geq 1-\delta$ for each $k\in[1:m]$. Then, $\mathbb{P}\left(\bigcap_{k=1}^{m}\mathcal{A}_{k}\right)\geq 1-\delta m$. *

Lemma 8** (Application of McDiarmid’s Inequality to the vector probability).**

Let ${P}\in\mathcal{P}(\mathcal{X})$ be any probability distribution and let $\hat{{P}}$ be its empirical estimation over a $n$-data set $\mathcal{S}_{n}$. Then, with probability at least $1-\delta$ over $\mathcal{S}_{n}$:

[TABLE]

Lemma 9** (Adaptation of Shamir et al., (2010)).**

Consider the encoder given by $q_{U|X}$. We have

[TABLE]

with

[TABLE]

for $\|\mathbf{P}_{X}-\mathbf{\hat{P}}_{X}\|_{2}$ not so small.

Lemma 10** (Adaptation of Shamir et al., (2010)).**

Let $\mathcal{U}$ a compact space. Consider the encoder $q_{U|X}$, then

[TABLE]

for $\max_{y}\|\mathbf{P}_{X|Y}(\cdot|y)-\mathbf{\hat{P}}_{X|Y}(\cdot|y)\|_{2}$ not so small

Lemma 11** (Shamir et al., (2010)).**

Let $n\geq a^{2}e^{2}$, then $\phi\left(\frac{a}{\sqrt{n}}\right)\leq\frac{a}{2}\frac{\log(n)}{\sqrt{n}}+\frac{e^{-1}}{\sqrt{n}}$.