Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothness

TL;DR

This paper develops a theoretical framework to quantify deep neural network generalization error based on data complexity and network smoothness, validated through experiments on image datasets.

Contribution

It introduces the cover complexity measure and inverse modulus of continuity to analyze neural network generalization, linking theoretical bounds with empirical observations.

Findings

Expected error scales with the square root of the number of classes.

Test loss correlates with neural network smoothness during training.

Network size affects smoothness, but dataset size does not.

Abstract

The accuracy of deep learning, i.e., deep neural networks, can be characterized by dividing the total error into three main types: approximation error, optimization error, and generalization error. Whereas there are some satisfactory answers to the problems of approximation and optimization, much less is known about the theory of generalization. Most existing theoretical works for generalization fail to explain the performance of neural networks in practice. To derive a meaningful bound, we study the generalization error of neural networks for classification problems in terms of data distribution and neural network smoothness. We introduce the cover complexity (CC) to measure the difficulty of learning a data set and the inverse of the modulus of continuity to quantify neural network smoothness. A quantitative bound for expected accuracy/error is derived by considering both the CC and neural network smoothness. Although most of the analysis is general and not specific to neural networks, we validate our theoretical assumptions and results numerically for neural networks by several data sets of images. The numerical results confirm that the expected error of trained networks scaled with the square root of the number of classes has a linear relationship with respect to the CC. We also observe a clear consistency between test loss and neural network smoothness during the training process. In addition, we demonstrate empirically that the neural network smoothness decreases when the network size increases whereas the smoothness is insensitive to training dataset size.

Tables

Table 1. Table 1: Cover complexity C ​ C ​ ( 𝒯 ) 𝐶 𝐶 𝒯 CC(\mathcal{T}) , best error ℰ ​ ( f ) ℰ 𝑓 \mathcal{E}(f) , and normalized error ℰ ​ ( f ) K ℰ 𝑓 𝐾 \frac{\mathcal{E}(f)}{\sqrt{K}} of different data sets. Different variants of data sets are used, including the original RGB or grey images (Original), grey images (Grey), and images extracted from a CNN (Conv). Images in CIFAR-100 have two variants: 100 categories (fine) and 20 categories (coarse). See  K for details.

Data Set	Variants	Input dim ($d$)	Output dim ($K$)	${\rho }_{𝒯}$	$CD\left(𝒯\right)$	$CC\left(𝒯\right)$	$ℰ\left(f\right)$	$\frac{ℰ\left(f\right)}{\sqrt{K}}$
MNIST	Original	784	10	.8480	.1053	1.442	.01	.0032
CIFAR-10	Original	3072	10	.8332	.0163	10.23	.45	.1423
CIFAR-10	Grey	1024	10	.8486	.0125	12.11	.53	.1676
CIFAR-10	Conv	1024	10	.9505	.0094	5.280	.18	.0569
SVHN	Original	3072	10	.9034	.0076	12.68	.49	.1550
SVHN	Grey	1024	10	.9117	.0084	10.48	.56	.1771
SVHN	Conv	1024	10	.9632	.0123	2.995	.23	.0727
CIFAR-100	Original (coarse)	3072	20	.8337	.0185	9.012	.62	.1386
CIFAR-100	Grey (coarse)	1024	20	.8541	.0132	11.08	.72	.1610
CIFAR-100	Conv (coarse)	1024	20	.9626	.0070	5.326	.40	.0894
COIL-20	Original	16384	20	.9176	.2385	.3453	.03	.0067
CIFAR-100	Original (fine)	3072	100	.8337	.0270	6.149	.73	.0730
CIFAR-100	Grey (fine)	1024	100	.8541	.0198	7.380	.81	.0810
CIFAR-100	Conv (fine)	1024	100	.9457	.0136	4.000	.52	.0520
COIL-100	Original	49152	100	.9430	.1944	.2930	.01	.0010

Table 2. Table 2: Comparison between c 𝑐 c -accuracy p c ​ ( f ) subscript 𝑝 𝑐 𝑓 p_{c}(f) with c = 0.5 𝑐 0.5 c=0.5 and the lower bounds for different training set sizes for the one-dimensional problem. Here δ 𝛿 \delta is indicated in Theorem  3.18 . The neural network is trained for 1000 iterations by the Adam optimizer.

$n$	$L_f^{max}$	${\rho }_{𝒯}$	$\delta$	$p_c\left(f\right)$	$h_{𝒯}\left(\delta \right)$	$1-\frac{\sqrt{d}}{\delta }\left(1-{\rho }_{𝒯}\right)$
10	.285	.972	.045	1.0	0.80	0.38
20	.246	.988	.041	1.0	1.00	0.69
40	.182	.994	.041	1.0	1.00	0.85
80	.127	.997	.038	1.0	1.00	0.92

Table 3. Table 3: The results in this table were obtained as follows: We used the ReLU as the activation function and Adam as the optimizer. Furthermore, we chose several different network structures and learning rates. We then trained the networks for 10000 iterations with a batch size of 300, and selected the best test error as the final result for each training procedure. Each of the columns i 𝑖 i - x ​ i ​ i 𝑥 𝑖 𝑖 xii stands for a specific choice of network architecture and learning rate, as described in Table 4

Data Set	Version	$i$	$ii$	$iii$	$iv$	$v$	$vi$	$vii$	$viii$	$ix$	$x$	$xi$	$xii$	Best Error
MNIST	Original	.02	.02	.05	.02	.02	.04	.02	.02	.04	.01	.02	.03	.01
CIFAR-10	Original	.47	.46	.52	.48	.46	.51	.47	.45	.50	.47	.45	.49	.45
CIFAR-10	Grey	.55	.55	.63	.55	.54	.62	.54	.53	.61	.54	.53	.59	.53
CIFAR-10	Conv	.18	.18	.19	.19	.18	.19	.18	.18	.18	.18	.18	.18	.18
SVHN	Original	.80	.59	.49	.80	.73	.60	.80	.69	.51	.80	.72	.64	.49
SVHN	Grey	.80	.64	.56	.80	.76	.66	.80	.64	.58	.80	.75	.66	.56
SVHN	Conv	.27	.23	.23	.31	.24	.23	.31	.24	.23	.69	.25	.24	.23
CIFAR-100	Original(coarse)	.64	.64	.69	.64	.63	.68	.63	.62	.67	.63	.62	.66	.62
CIFAR-100	Grey(coarse)	.74	.73	.79	.74	.73	.78	.74	.72	.78	.74	.72	.77	.72
CIFAR-100	Conv(coarse)	.40	.41	.45	.41	.41	.44	.40	.41	.44	.41	.40	.43	.40
COIL-20	Original	.08	.05	.05	.07	.06	.05	.08	.05	.05	.07	.05	.03	.03
CIFAR-100	Original(fine)	.75	.75	.82	.75	.74	.81	.74	.73	.80	.75	.73	.79	.73
CIFAR-100	Grey(fine)	.83	.83	.90	.83	.83	.90	.82	.81	.88	.83	.81	.87	.81
CIFAR-100	Conv(fine)	.53	.54	.64	.53	.54	.63	.52	.52	.61	.53	.52	.59	.52
COIL-100	Original	.02	.01	.01	.03	.02	.01	.03	.02	.01	.02	.02	.01	.01

Table 4. Table 4: Detailed setup for each case.

	${10}^{-3}$	${10}^{-4}$	${10}^{-5}$
$[$Input 256 256 Output$]$	$i$	$ii$	$iii$
$[$Input 256 256 256 Output$]$	$iv$	$v$	$vi$
$[$Input 512 512 Output$]$	$vii$	$viii$	$ix$
$[$Input 512 512 512 Output$]$	$x$	$xi$	$xii$

Equations

$$D=[0,1{]}^d,$$

$$T=2^{\{1,2,\cdots ,K\}}\backslash \{\varnothing \}$$

$$\tau_{T}=\left\{\bigcup_{k\in I}U_{k}\Bigg{|}I\subseteq T\right\},$$

$$\forall x\in D\forall k\in f(x)\exists \delta >0s.t.k\in f(y)if\Vert y-x\Vert <\delta .$$

$$tag:D\to T$$

$$\forall x\in D\forall k\in tag(x)\exists \delta >0s.t.k\in tag(y)if\Vert y-x\Vert <\delta .$$

$$\begin{array}{rl}{\delta }_0=\inf \{& \Vert x-y\Vert |tag(x)\ne tag(y),and\\ & tag(x),tag(y)\in \{\{1\},\{2\},\cdots ,\{K\}\}\}.\end{array}$$

$$\mathcal{T}=\{x_1,x_2,\dots ,x_n\}\subseteq D.$$

$$h_{\mathcal{T}}^{\mu }(r):=\mu \left(D\cap \bigcup _{x_i\in \mathcal{T}}B(x_i,r)\right),$$

$$\rho (\mathcal{T},\mu ):=\frac{1}{\sqrt{d}}{\int }_0^{\sqrt{d}}{h}_{\mathcal{T}}^{\mu }(r)dr.$$

$${\rho }_{\mathcal{T}}:=\rho (\mathcal{T},\mu ).$$

$$CD(\mathcal{T}):=\frac{1}{K}\sum _i\rho ({\mathcal{T}}_i,{\mu }_i)-\frac{1}{K(K-1)}\sum _{i\ne j}\rho ({\mathcal{T}}_i,{\mu }_j),$$

$${\mathcal{T}}_i:=\{x\in \mathcal{T}|i\in tag(x)\},{\mu }_i(A):=\frac{\mu (A\cap D_i)}{\mu (D_i)}$$

$$CC(\mathcal{T}):=\frac{1-{\rho }_{\mathcal{T}}}{CD(\mathcal{T})}$$

$$h_{\mathcal{T}}^{\mu }(r)\ge 1-\frac{\sqrt{d}}{r}(1-{\rho }_{\mathcal{T}}),0<r\le \sqrt{d}.$$

$$m=O\left(\frac{d+1}{ϵ}\ln \frac{d+1}{ϵ}+\frac{1}{ϵ}\ln \frac{1}{\eta }\right),$$

$${\rho }_{\mathcal{T}}\ge \varrho (ϵ)$$

$$\hat{f}:x\mapsto \frac{e^{f(x)}}{{\sum }_i{e}^{f_i(x)}}$$

$$\exists i_{max}\in tag(x),s.t.f_{i_{max}}(x)>c.$$

$$p_c(f):=\frac{\mu (H_c^f)}{\mu (D)}=\mu (H_c^f),$$

$$p_c^{\mathcal{T}}:=\frac{{\rho }_{{\tilde{\mathcal{T}}}_c}}{{\rho }_{\mathcal{T}}},$$

$$p(f):=\frac{\mu (H^f)}{\mu (D)}=\mu (H^f),$$

$$\forall ϵ>0,\exists \delta >0,s.t.\Vert f(x)-f(y){\Vert }_{\infty }<ϵwhen\Vert x-y\Vert <\delta .$$

$${\varphi }_i(x)=\sigma (W_{i}x+b_i),1\le i\le l-1,$$

$$f(x)=softmax(W_l({\varphi }_{l-1}\circ \cdots \circ {\varphi }_1(x))+b_l),$$

$$W_i\in {\mathbb{R}}^{n_i\times n_{i-1}},b_i\in {\mathbb{R}}^{n_i},1\le i\le l,$$

$${\delta }_f(ϵ)\ge \frac{ϵ}{Lip(f)}\ge \frac{ϵ}{\Vert W_1{\Vert }_2\cdots \Vert W_l{\Vert }_2\cdot Lip(softmax)},$$

$$p_{c_2}(f)\ge 1-\frac{\sqrt{d}}{\delta }(1-p_{c_1}^{\mathcal{T}}{\rho }_{\mathcal{T}}),$$

$$L_f^{max}:=\underset{1\le i\le n}{\max }\ell (f(x_i),k_i)=-\underset{1\le i\le n}{\min }\ln (f_{k_i}(x_i))<-\ln c,$$

$$p_c(f)\ge 1-\frac{\sqrt{d}}{\delta }(1-{\rho }_{\mathcal{T}}),$$

Full text

††journal:

1 Introduction

In the last 15 years, deep learning, i.e., deep neural networks (NNs), has been used very effectively in diverse applications, such as image classification (Krizhevsky et al., 2012), natural language processing (Maas et al., 2013), and game playing (Silver et al., 2016). Despite this remarkable success, our theoretical understanding of deep learning is lagging behind. The accuracy of NNs can be characterized by dividing the expected error into three main types: approximation (also called expressivity), optimization, and generalization (Bottou and Bousquet, 2008; Bottou, 2010), see Fig. 1. The well-known universal approximation theorem was obtained by Cybenko (1989) and Hornik et al. (1989) almost three decades ago stating that feed-forward neural nets can approximate essentially any function if their size is sufficiently large. In the past several years, there have been numerous studies that analyze the landscape of the non-convex objective functions, and the optimization process by stochastic gradient descent (SGD) (Lee et al., 2016; Liao and Poggio, 2017; Lu et al., 2018; Allen-Zhu et al., 2018b; Du et al., 2018; Lu et al., 2019). Whereas there are some satisfactory answers to the problems of approximation and optimization, much less is known about the theory of generalization, which is the focus of this study.

The classical analysis of generalization is based on controlling the complexity of the function class, i.e., model complexity, by managing the bias-variance trade-off (Friedman et al., 2001). However, this type of analysis is not able to explain the small generalization gap between training and test performance of neural networks learned by SGD in practice, considering the fact that deep neural networks often have far more model parameters than the number of samples they are trained on, and have sufficient capacity to memorize random labels (Neyshabur et al., 2014; Zhang et al., 2016). To explain this phenomenon, several approaches have been recently developed by many researchers. The first approach is characterizing neural networks with some other low “complexity” instead of the traditional Vapnik-Chervonenkis (VC) dimension (Bartlett et al., 2017b) or Rademacher complexity (Bartlett and Mendelson, 2002), such as path-norm (Neyshabur et al., 2015), margin-based bounds (Sokolić et al., 2017; Bartlett et al., 2017a; Neyshabur et al., 2017b), Fisher-Rao norm (Liang et al., 2017), and more (Neyshabur et al., 2019; Wei and Ma, 2019). The second approach is to analyze some good properties of SGD or its variants, including its stability (Hardt et al., 2015; Kuzborskij and Lampert, 2017; Gonen and Shalev-Shwartz, 2017; Chen et al., 2018), robustness (Sokolic et al., 2016; Sokolić et al., 2017), implicit biases/regularization (Poggio et al., 2017; Soudry et al., 2018; Gunasekar et al., 2018; Nagarajan and Kolter, 2019b), and the structural properties (e.g., sharpness) of the obtained minimizers (Keskar et al., 2016; Dinh et al., 2017; Zhang et al., 2018). The third approach relies on overparameterization, e.g., sufficiently overparameterized networks can learn the ground truth with a small generalization error using SGD from random initialization (Li and Liang, 2018; Allen-Zhu et al., 2018a; Arora et al., 2019; Cao and Gu, 2019). There are also other approaches, such as compression (Arora et al., 2018; Baykal et al., 2018; Zhou et al., 2018; Cheng et al., 2018), Fourier analysis (Rahaman et al., 2018; Xu et al., 2019), “double descent” risk curve (Belkin et al., 2018), PAC-Bayesian framework (Neyshabur et al., 2017b; Nagarajan and Kolter, 2019a), and information bottleneck (Shwartz-Ziv and Tishby, 2017; Saxe et al., 2019).

However, most theoretical bounds fail to explain the performance of neural networks in practice (Neyshabur et al., 2017a; Arora et al., 2018). To get non-vacuous and tight enough bounds to be practically meaningful, some problem-specific factors should be taken into consideration, such as the low complexity (i.e., data-dependent analysis) (Dziugaite and Roy, 2017; Kawaguchi et al., 2017), or properties of the trained neural networks (Sokolić et al., 2017; Arora et al., 2018; Wei and Ma, 2019). In this study, to achieve a practically meaningful bound, our analysis relies on the data distribution and the smoothness of the trained neural network. The analysis proposed in this study provides guarantees on the generalization error, and theoretical insights to guide the practical application.

As shown in Fig. 1, the optimization error is correlated with the loss value (for notation simplicity, the term “loss” indicates “empirical loss”), while the approximation error depends on the network size. In addition, a small loss requires a sufficient approximation ability, i.e., a large network size, which in turn leads to a small approximation error. If we assume a sufficiently small loss, which usually holds in practice, then the expected error mainly depends on the generalization error. Hence, we study the expected error/accuracy directly. In particular, we propose a mathematical framework to analyze the expected accuracy of neural networks for classification problems. We introduce the concepts of total cover (TC), self cover (SC), mutual cover (MC) and cover difference (CD) to represent the data distribution, and then we use the concept of cover complexity (CC) as a measure of the complexity of classification problems. On the other hand, the smoothness of a neural network $f$ is characterized by the inverse of the modulus of continuity $\delta_{f}$. Because computing $\delta_{f}$ is not tractable in general, we propose an estimation using the spectral norm of the weight matrices of the neural network. The main terminologies are illustrated in Fig 2. By combining the properties of the data distribution and the smoothness of neural networks, we derive a lower bound for the expected accuracy, i.e., an upper bound for the expected classification error.

Subsequently, we test our theoretical bounds on several data sets, including MNIST (LeCun et al., 1998), CIFAR-10 (Krizhevsky and Hinton, 2009), CIFAR-100 (Krizhevsky and Hinton, 2009), COIL-20 (Nene et al., 1996b), COIL-100 (Nene et al., 1996a), and SVHN (Netzer et al., 2011). Our numerical results not only confirm our theoretical bounds, but also provide insights into the optimization process and the learnability of neural networks. In particular, we find that:

The best accuracy that can be achieved in practice (i.e., optimized by stochastic gradient descent) by fully-connected networks is approximately linear with respect to the cover complexity of the data set.

• 2.

The trend of the expected accuracy is consistent with the smoothness of the neural network, which provides a new “early stopping” strategy by monitoring the smoothness of the neural network.

• 3.

The neural network smoothness decreases when the network depth and width increases, with the effects of depth more significant than that of width.

• 4.

The neural network smoothness is insensitive to the training dataset size, and is bounded from below by a positive constant. This point makes our theoretical result (Theorem 3.20) specifically pertinent to deep neural networks.

The paper is organized as follows. After setting up notation and terminology in Section 2, we present the main theoretical bounds for the accuracy based on the data distribution and the smoothness of neural networks in Section 3, while all proofs are deferred to the appendix. In Section 4, we provide the numerical results for several data sets. In Section 5 we include a discussion, and in Section 6 we summarize our findings.

2 Preliminaries

Before giving the main results, we introduce the necessary notation and terminology. Without loss of generality, we assume that the space we need to classify is

[TABLE]

where $d$ is the dimensionality, and the points in this space are classified into $K$ categories, i.e., there are $K$ labels $\{1,2,\cdots,K\}$. We denote the probability measure on $D$ by $\mu$, i.e., for a measurable set $A\subseteq D$, $\mu(A)$ is the probability of a random sample belonging to $A$.

2.1 Ideal label function

For the problem setup, we assume that every sample has at least one true label, and one sample may have multiple true labels. Taking image classification as an example, each image has at least one correct label. A fuzzy image or an image with more than one object in it may have multiple possible correct labels, and as long as the prediction is one of these labels, we consider the prediction to be correct.

It is intuitive that when two samples are close enough, they should have similar labels, which means that the ideal label function should be continuous. The continuity of a mapping depends on the topology of both domain and image space. For the domain of the ideal label function, we choose the standard topology induced by the Euclidean metric. As for the topology of the image space, we define it as follows. We first define the label set and the topology on it.

Definition 2.1** (Topology).**

Let

[TABLE]

be the label set. Define the topology on $T$ to be

[TABLE]

where $U_{k}=\{V\in T|V\supseteq k\}$ for $k\in T$, and thus $(T,\tau_{T})$ constitutes a topological space.

Analogous to the Euclidean metric topology, $U_{k}$ is viewed as the open “ball” centered at $k$, and arbitrary unions of the “balls” $U_{k}$ are defined as the open sets, see A for an example. With this choice of the topology, a function $f:D\to T$ is continuous if and only if

[TABLE]

Next we give the definition of the ideal label function according to this topological space.

Definition 2.2** (Ideal label function).**

An idea label function is a continuous function

[TABLE]

where $D$ is equipped with the Euclidean metric topology and $T$ with the topology $\tau_{T}$ from Definition 2.1. This continuity holds if and only if

[TABLE]

Eq. (1) means that two neighboring points would have some common labels. Based on the topological space defined above, it is easy to show that Eq. (1) is equivalent to continuity. The reason why we consider a multi-label setup for classification problems is that it allows for the continuity property in Eq. (1), which is impossible in the setup of a single label set, unless the label function is constant. In addition, the multi-label setup introduces a smooth transition, i.e., a buffer domain, between two domains of different labels, while the transition is sharp in the single label setup. In the following proposition, we show that if two samples are close enough, they must share at least one common label.

Proposition 2.3** (Separation gap).**

$\exists\delta>0,\ s.t.\ tag(x)\cap tag(y)\neq\varnothing\ when\ \|x-y\|<\delta.$* We denote the supremum of $\delta$ as the separation gap $\delta_{0}$, which is used in the sequel.*

Proof.

The proof can be found in  B. ∎

To understand the geometric interpretation of $\delta_{0}$, we consider the following special case: the label of each sample is either a single label set, such as $\{1\}$, or the full label set $\{1,2,\cdots,K\}$ if it is not uniquely identifiable.

Proposition 2.4** (Geometric interpretation of separation gap).**

If the label of each sample is either a single label set or the full label set $\{1,2,\cdots,K\}$, then $\delta_{0}$ is the smallest distance between two different single label points, i.e.,

[TABLE]

Proof.

The proof can be found in  C. ∎

2.2 Cover complexity of data set

In this subsection, we introduce a quantity to measure the difficulty of learning a training data set

[TABLE]

First, we give some notations and propositions.

With the measure $\mu$, the probability of the neighborhood of the training set with radius of $r$ is defined as

[TABLE]

where $B(x_{i},r)$ is the open ball centered at $x_{i}$ with radius of $r$, see Fig. 2A. Obviously, $h_{\mathcal{T}}^{\mu}(r)$ is a monotone non-decreasing function, $h_{\mathcal{T}}^{\mu}(0)=0$ (since $B(x,0)=\varnothing$), and $h_{\mathcal{T}}^{\mu}(r)=1$ when $r>\sqrt{d}$, see Fig. 2B. To represent the global behavior of $h_{\mathcal{T}}^{\mu}(r)$, we use the integral of $h_{\mathcal{T}}^{\mu}(r)$ with respect to $r$:

[TABLE]

Hence, $\rho(\mathcal{T},\mu)$ considers both the number and location of the data points, and also the probability distribution of the space. The value $\rho(\mathcal{T},\mu)$ is larger if the number of data points is increased and also if the probability distribution is more concentrated around $\mathcal{T}$, which we call the “coverability” of $\mathcal{T}$. We can increase $\rho(\mathcal{T},\mu)$ by adding more data points or redistribute their locations. Next, we introduce the formal definition for the “coverability”.

Definition 2.5** (Coverability).**

Let $\mathcal{T}$ be a data set from a domain $D$ with probability measure $\mu$. We define the following for the coverability of $\mathcal{T}$.

1. (i)

The total cover (TC) is

[TABLE]

Thus, $0\leq\rho_{\mathcal{T}}\leq 1$. 2. (ii)

The cover difference (CD) is

[TABLE]

where $K$ is the number of categories, and $\mathcal{T}_{i}\subset\mathcal{T}$ and $\mu_{i}$ represent the subset and probability measure of the label $i$ respectively, i.e.,

[TABLE]

with $D_{i}:=\{x\in D|i\in tag(x)\}$. Here, $\rho(\mathcal{T}_{i},\mu_{i})$ is called self cover (SC), and $\rho(\mathcal{T}_{i},\mu_{j})$ is called mutual cover (MC). 3. (iii)

The cover complexity (CC) is

[TABLE]

for $CD(\mathcal{T})\neq 0$.

Remark 2.6**.**

CD is defined as the difference between the mean of SC and the mean of MC, since each category occurs with the same probability ($\sim 1/K$) in the data sets mostly used in practice. If there are some categories occurring more frequently than others, then it is straightforward to extend this definition by using the mean weighted by the probability of each category.

In image classification, the dimension of the image space is very high, and thus the data points are quite sparse. However, due to the fact that images actually live on a manifold of low dimension, the probability density around $\mathcal{T}$ is actually high, which makes the TC to be meaningful. In our next result, we derive a lower bound of $h_{\mathcal{T}}^{\mu}(r)$ by $\rho_{\mathcal{T}}$.

Proposition 2.7**.**

Let $\mathcal{T}$ be a data set. $h_{\mathcal{T}}^{\mu}(r)$ and $\rho_{\mathcal{T}}$ are defined as above. Then we have

[TABLE]

Proof.

The proof can be found in  D. ∎

From this proposition, we know that for a fixed $r$, $h_{\mathcal{T}}^{\mu}(r)$ is close to 1 if $\rho_{\mathcal{T}}$ is large enough. However, the probability distribution is usually given in practice, and we can only control the number of samples. The following theorem shows that $\rho_{\mathcal{T}}$ can be arbitrary close to 1 when enough samples are available.

Theorem 2.8**.**

Let $\mathcal{T}$ be a data set of size $n$ drawn according to $\mu$. Then there exists a non-increasing function $\varrho(\epsilon)$ satisfying $\lim\limits_{\epsilon\to 0}\varrho(\epsilon)=1$, and for any $0<\eta,\epsilon\leq\frac{1}{2}$, there exists an

[TABLE]

such that

[TABLE]

holds with probability at least $1-\eta$ when $n\geq m$.

Proof.

The proof and some other results regarding TC can be found in E. ∎

The reason why CD is introduced is that TC does not consider the labels of the data points. However, data points of the same label should be clustered in an easily learnable data set. $CD(\mathcal{T})$ is the difference of self cover and mutual cover, which considers the distributions of each label. By normalizing TC with CD, the cover complexity $CC(\mathcal{T})$ is able to measure the difficulty of learning a data set. The difficulty of a problem should be translation-independent and scale-independent. It is easy to see that $CC(\mathcal{T})$ is independence of translation, and the following proposition shows that it is also scale-independent.

Proposition 2.9** (Scale independence).**

$CC(\mathcal{T})$* is scale-independent, i.e., if all the data points are scaled by the same factor less than 1, then $CC(\mathcal{T})$ is unchanged.*

Proof.

The proof can be found in  F. ∎

2.3 Setup for accuracy analysis

The setup for accuracy analysis is as follows.

Definition 2.10**.**

If $f:D\rightarrow\mathbb{R}^{K}$ is a continuous mapping, then the mapping

[TABLE]

is still continuous, where $f_{i}(x)$ represents the i-th component of $f(x)$, and $e^{f(x)}\in\mathbb{R}^{K}$ is the exponential function applied componentwise to $f(x)$. We have $\hat{f}_{i}(x)>0$, and $\sum_{i}\hat{f}_{i}(x)=1$. For convenience, we directly consider the case that $f_{i}(x)>0$ and $\sum f_{i}(x)=1$, and we call such mapping the normalized continuous positive mapping.

Remark 2.11**.**

A neural network with softmax nonlinearity is a normalized continuous positive mapping.

Different from the accuracy usually used in classification problems, we define a stronger accuracy called $c$-accuracy as follows.

Definition 2.12** ($c$-accuracy at $x$).**

Let $f$ be a normalized continuous positive mapping. For $0.5\leq c<1$, we say that $f$ is $c$-accurate at point $x$ if

[TABLE]

Definition 2.13** ($c$-accuracy on $D$).**

Let $f$ be a normalized continuous positive mapping. The $c$-accuracy of $f$ on a sample space $D$ is defined as

[TABLE]

where $H_{c}^{f}:=\{x\in D|f$ is $c$-accurate at x$$\}.

Definition 2.14** ($c$-accuracy on $\mathcal{T}$).**

Let $f$ be a normalized continuous positive mapping. The $c$-accuracy of $f$ on a data set $\mathcal{T}$ is defined as

[TABLE]

where $\widetilde{\mathcal{T}}_{c}:=\{x\in\mathcal{T}|f$ is $c$-accurate at $x\}$, and $\rho_{\widetilde{\mathcal{T}}_{c}}$ and $\rho_{\mathcal{T}}$ are the TC of $\widetilde{\mathcal{T}}_{c}$ and $\mathcal{T}$, respectively.

Definition 2.15** (Expected accuracy).**

Let $f$ be a normalized continuous positive mapping. The expected accuracy of $f$ on a sample space $D$ is defined as

[TABLE]

where $H^{f}:=\{x\in D|\exists i_{max}\in tag(x),\ s.t.\ f_{i_{max}}(x)>f_{i}(x)\ \forall 1\leq i\leq K,\ i\neq i_{max}\}$.

We note that the $c$-accuracy of $f$ on $D$ represents the expected $c$-accuracy, and the $c$-accuracy of $f$ on $\mathcal{T}$ represents the empirical $c$-accuracy.

Finally, we define a non-decreasing function $\delta_{f}$ to describe the smoothness of $f$.

Definition 2.16** (Smoothness).**

Let $f:D\to\mathbb{R}^{K}$ be a continuous mapping. Then $f$ is uniformly continuous due to the compactness of $D$, i.e.

[TABLE]

We denote the supremum of $\delta$ satisfying the above requirement by $\delta_{f}(\epsilon)$. It is easy to see that $\delta_{f}(\epsilon)$ is equal to the inverse of the modulus of continuity of $f$.

For low dimensional problems, we can directly compute $\delta_{f}$ by brute force. However, for high dimensional problems, it is intractable to compute $\delta_{f}$, and thus we give the following lower bound of $\delta_{f}$ for a fully-connected ReLU-network $f$ with softmax as the activation function in the last layer, which is also the main network structure considered through this work.

A fully-connected neural network is defined as follows:

[TABLE]

[TABLE]

[TABLE]

where $n_{i}$ is the number of neurons in the layer $i$ ($n_{0}=d$ and $n_{l}=K$), and $\sigma$ is the activation function. Then for the ReLU activation function, we have

[TABLE]

where $\|\cdot\|_{2}$ is the spectral norm, $Lip(f)$ and $Lip(softmax)$ represent the Lipschitz constants of $f$ and $softmax$ mapping from $(D,\|\cdot\|_{2})$ to $(\mathbb{R}^{K},\|\cdot\|_{\infty})$, respectively. $Lip(softmax)$ is a constant less than $1/2$, and thus is ignored in our numerical examples. We note that although the lower bound of $\delta_{f}(\epsilon)$ depends exponentially on the neural net depth, $\delta_{f}(\epsilon)$ itself does not necessarily scale exponentially with the network depth.

3 Lower bounds for the expected accuracy

In this section, we present a theoretical analysis of the lower bound for the expected accuracy as well as an upper bound for the expected error.

Proposition 3.17**.**

Let $f$ be a normalized continuous positive mapping. Suppose that $\mathcal{T}$ is a single label training set, i.e. $tag(\mathcal{T})\subseteq\{\{1\},\{2\},\cdots,\{K\}\}$. For any $0.5\leq c_{2}<c_{1}\leq 1$, we have

[TABLE]

where $\delta=\min(\delta_{0},\delta_{f}(c_{1}-c_{2}))$.

Proof.

The proof can be found in  G. ∎

Proposition 3.17 shows that the expected $c_{2}$-accuracy of $f$ can be bounded by the empirical $c_{1}$-accuracy and the TC of the training set. We can see that $p_{c_{2}}(f)$ tends to 1 when $\rho_{\mathcal{T}}$ and $p_{c_{1}}^{\mathcal{T}}$ tend to 1. Next we derive a bound for the accuracy by taking into account the loss function.

Theorem 3.18** (Lower bound of $c$-accuracy).**

Let $f$ be a normalized continuous positive mapping. Suppose that $\mathcal{T}=\{x_{1},\cdots,x_{n}\}$ is a single label training set, and $tag(x_{i})=\{k_{i}\}$. For any $0.5\leq c<1$, if the maximum cross entropy loss

[TABLE]

then we have

[TABLE]

where $\ell$ is the cross entropy loss that $\ell(f(x),k)=-\ln(f_{k}(x))$, $\delta=\min(\delta_{0},\delta_{f}(e^{-L_{f}^{max}}-c))$, and $\delta_{0}$ is defined in Proposition 2.3.

Proof.

The proof can be found in  H. ∎

Theorem 3.18 reveals that the expected accuracy is related to the total cover $\rho_{\mathcal{T}}$, separation gap $\delta_{0}$, neural network smoothness $\delta_{f}$, and loss value $L_{f}^{max}$. We will show numerically in Section 4 that $\delta_{f}(e^{-L_{f}^{max}}-c)$ increases first and then decreases during the training of neural networks. The following theorem states that the maximum value of $\delta_{f}(e^{-L_{f}^{max}}-c)$ is bounded by the empirical separation gap.

Theorem 3.19** (Empirical separation gap).**

Let $f$ be a normalized continuous positive mapping. Suppose that $\mathcal{T}=\{x_{1},\cdots,x_{n}\}$ is a single label training set. For any $0.5\leq c<1$, if $L_{f}^{max}<-\ln c$, then we have

[TABLE]

where

[TABLE]

is called the empirical separation gap, i.e., the smallest distance between two differently labeled training points.

Proof.

The proof can be found in  I. ∎

Besides the upper bound, the lower bound of $\delta_{f}$ is also important to the accuracy. We have observed that in practice the low bound of $\delta_{f}$ exists (Figs. 5, 6, 7 and 8), which indicates the existence of $\kappa$ in the following theorem. Based on this observation, we have the following theorem for the accuracy.

Theorem 3.20** (Lower bound of accuracy).**

Assume that there exists a constant $\kappa>0$, such that

[TABLE]

holds for any single label training set $\mathcal{T}$ and a corresponding suitable trained network $f$ on $\mathcal{T}$ such that $L_{f}^{max}(\mathcal{T})<\ln 2$, then we have the following conclusions for the expected accuracy $p(f)$ and the expected error $\mathcal{E}(f)=1-p(f)$:

1. (i)

*with the same condition of Theorem 3.18, *

$p(f)\geq 1-\frac{\sqrt{d}}{\delta}(1-\rho_{\mathcal{T}})$, 2. (ii)

$\mathcal{E}(f)\leq\frac{\sqrt{d}\cdot(1-\rho_{\mathcal{T}})}{\min(\delta_{0},\kappa\delta_{\mathcal{T}})}=\alpha(\mathcal{T})\cdot CC(\mathcal{T}),$** 3. (iii)

$\lim\limits_{\rho_{\mathcal{T}}\to 1}p(f)=1,$**

where $\delta=\min(\delta_{0},\delta_{f}(e^{-L_{f}^{max}}-c))$, and $\alpha(\mathcal{T})=\frac{\sqrt{d}\cdot CD(\mathcal{T})}{\min(\delta_{0},\kappa\delta_{\mathcal{T}})}$.

Proof.

(i) is the conclusion of Theorem 3.18. The proof of (ii) and (iii) can be found in  J. ∎

Remark 3.21**.**

We have the following remarks for this main theorem:

In (ii), we rewrite $\frac{\sqrt{d}\cdot(1-\rho_{\mathcal{T}})}{\min(\delta_{0},\kappa\delta_{\mathcal{T}})}$ to emphasize $CC(\mathcal{T})$ by introducing a coefficient term $\alpha(\mathcal{T})$. Although this seems artificial, our numerical experiments empirically show that in practice the linear scaling between $\mathcal{E}(f)$ and $CC(\mathcal{T})$ is indeed satisfied.

• 2.

This theorem is not specific to neural networks, but rather holds for any trained model $f$ satisfying the assumptions required in the theorem. While the assumptions are not true for a general machine learning algorithm, we show numerically that in practice neural networks satisfy the assumptions.

• 3.

The current proof of the theorem relies on the assumption of the maximum cross entropy $L_{f}^{max}$, which could be hard to satisfy in practice, since it will be large even if only a single training sample is misclassified. However, the assumption of the maximum cross entropy is possible to be relaxed according to our experiments.

Here, the cover complexity $CC(\mathcal{T})$ consists of two parts, one represents the richness of the whole training set while the other part describes the degree of separation between different labeled subsets. As for $\alpha(\mathcal{T})$, both the denominator and numerator seem to have a positive correlation with respect to separation level. What we wish is that $\alpha(\mathcal{T})$ is almost close to a constant with high probability and the expected error $\mathcal{E}(f)$ is mainly determined by $CC(\mathcal{T})$, which approximately represents the complexity level of the data set. We will provide more information in detail in the section concerning the numerical results.

4 Numerical results

In this section, we use numerical simulations to test the accuracy of neural networks in terms of the data distribution (cover complexity), and neural network smoothness. In addition, we study the effects of the network size and training dataset size on the smoothness. The codes are published in GitHub (https://github.com/jpzxshi/generalization).

4.1 Data distribution

In this subsection, we explore how $CC(\mathcal{T})$ affects the expected error $\mathcal{E}(f)$. In our experiments, we test several data sets, including MNIST (LeCun et al., 1998), CIFAR-10 (Krizhevsky and Hinton, 2009), CIFAR-100 (Krizhevsky and Hinton, 2009), COIL-20 (Nene et al., 1996b), COIL-100 (Nene et al., 1996a), SVHN (Netzer et al., 2011). In addition to the original data set, we also create some variants: (1) the images of grey color, (2) the images extracted from a convolutional layer after training the original data set using a convolutional neural network (CNN), (3) combine several categories into one category to reduce the number of total categories, see Table 1 and details in K.

For a training data set $\mathcal{T}$, we estimate $h_{\mathcal{T}}^{\mu}(r)$ by the proportion of the test data points within the balls with radius $r$ centered at training data points, i.e.,

[TABLE]

and then $\rho_{\mathcal{T}}$ is obtained by Definition 2.5. Similarly, we estimate $CD(\mathcal{T})$ and then compute $CC(\mathcal{T})$. Next for each data set, we train fully-connected neural networks with different hyperparameters, and record the best error we observed, see the details in K. The cover complexity and the best error for each data set are shown in Table 1.

These data sets are divided into three groups according to their output dimensions. For each group of the same output dimension, the error is almost linearly correlated with $CC(\mathcal{T})$, see Fig. 3A, regardless of the input dimension. In addition, we find that all the cases collapse into a single line when normalizing the error $\mathcal{E}(f)$ by a factor of $1/\sqrt{K}$, see Fig. 3B.

It is noteworthy that the $CC(\mathcal{T})$ of convolutional variants of data sets is much smaller than that of the original data sets, and hence the expected accuracy increases. The results confirm the importance of data distribution.

Next, we consider the most difficult data set, i.e., data with random labels. We choose MNIST and then assign each image a random label. We repeat this process 50 times, and compute each $CC(\mathcal{T})$. The distribution of $|CC(\mathcal{T})|$ is shown in Fig. 4. The smallest $|CC(\mathcal{T})|$ is $\sim$300, which is much larger than that of the original data sets with $CC(\mathcal{T})<20$. This extreme example again confirms that $CC(\mathcal{T})$ is a proper measure of the difficulty of classifying a data set.

4.2 Neural network smoothness

In this subsection, we will investigate the relationship between the neural network smoothness $\delta_{f}(e^{-L_{f}^{max}}-c)$ and the accuracy, and the effects of network size (depth and width) on the smoothness. We first show results for one- and two-dimensional problems, where $\delta_{f}(e^{-L_{f}^{max}}-c)$ can be computed accurately by brute force. Subsequently, we consider the high dimensional setting of the MNIST data set, where we estimate $\delta_{f}(e^{-L_{f}^{max}}-c)$ by Eq. (3).

4.2.1 One- and two-dimensional problems

We first consider a one-dimensional case and a two-dimensional case. For the one-dimensional case, we choose the sample space $D=[0,1]$, $K=2$, and the ideal label function as

[TABLE]

with separation gap $\delta_{0}=0.1$. We use $n$ equispaced points ($n\geq 4$ is an even number) on $D\setminus(0.5-\frac{\delta_{0}}{2},0.5+\frac{\delta_{0}}{2})$ as the training set, i.e., $\mathcal{T}=\mathcal{T}_{1}\cup\mathcal{T}_{2}$, where

[TABLE]

We choose 10000 equispaced points on $D$ as the test data.

For the two-dimensional case, we choose the sample space $D=[0,1]^{2}$, $K=2$, and the ideal label function as

[TABLE]

with $\delta_{0}=0.1$. For the training set, we first choose $n=m^{2}$ equispaced points, i.e., $\mathcal{T}=\{0,\frac{1}{m-1},\frac{2}{m-1},\dots,\frac{m-2}{m-1},1\}^{2}$, and then remove the points with label $\{1,2\}$ to ensure that all samples are of single label. We choose $1000^{2}$ equispaced points on $D$ as the test data.

In our experiments, we use a 3-layer fully-connected NN with ReLU activation and 30 neurons per layer. The neural network is trained for 1000 iterations by the Adam optimizer (Kingma and Ba, 2015) for the one-dimensional problem, and 2000 iterations for the two-dimensional problem. For the one-dimensional problem, the $c$-accuracy $p_{c}(f)$ with $c=0.5$ and lower bounds for different numbers of training points are listed in Table 2. We can see that the bounds become tighter when $n$ is larger.

During the training process of the neural network, the test loss first decreases and then increases, while $\delta_{f}$ first increases and then decreases, see Fig. 5A for the one-dimensional problem ($n=20$) and Fig. 5B for the two-dimensional problem ($n=400$). $\delta_{f}$ is bounded by $\delta_{\mathcal{T}}/2$, as proved in Theorem 3.19. We also observe that the trends of test loss and $\delta_{f}$ coincide, and thus we should stop the training when $\delta_{f}$ begins to decrease to prevent overfitting.

4.2.2 High-dimensional problem

In the high-dimensional problem of MNIST, we consider the average loss $L_{f}$ instead of the maximum loss $L_{f}^{max}$, which is very sensitive to extreme points. As shown in Eq. (3), we use the following quantity to bound $\delta_{f}(e^{-L_{f}}-c)$:

[TABLE]

Because we use the $c$-accuracy to approximate the true accuracy, for the classification problems with two categories, $p_{0.5}(f)$ and $p(f)$ are equivalent. However, they are not equal for problems with more than two categories, where the best $c$ depends on the properties of the data set, such as the easiness of learning to classify the data set. If the data set is easy to classify, such as MNIST, the best $c$ should be close to 1. In our example, we choose $c=0.9$. We train MNIST using a 3-layer fully-connected NN with ReLU activation and 100 neurons per layer for 100 epochs. In Fig. 6, we can also see the consistency between the test loss and neural network smoothness, as we observed in the low-dimensional problems.

4.2.3 Effects of the network size and training dataset size on the smoothness

We have demonstrated that network smoothness $\delta_{f}(e^{-L_{f}^{max}}-c)$ is an important factor to the accuracy. Next, we investigate the effects of network size (depth and width) on the smoothness for binary classification problems, which are explained as follows. We consider the one-dimensional sample space $D=[0,1]$, and choose $n$ equispaced points on $D$ as the training data locations. To avoid the effects of the choice of target true functions, we always repeat experiments with different target functions, and in each experiment we generate a random target function. Specifically, to generate a random target function, we first sample two random functions $g_{1}(x)$ and $g_{2}(x)$ from a Gaussian process with the radial basis function kernel of a length scale 0.2, and then assign a point $x$ as category 1 if $g_{1}(x)>g_{2}(x)$, otherwise assign this point as category 2. When training neural networks, we monitor the value of $\delta_{f}(e^{-L_{f}^{max}}-c)$ and stop the training once $\delta_{f}$ begins to decrease as shown in Figs. 5 and 6. We first choose the dataset size $n=10$, and we show that the normalized smoothness $\delta_{f}/\delta_{T}$ decreases as the network depth or width increases (Fig. 7). We also show that the effects of depth is more significant than that of width.

Our main theorem (Theorem 3.20) requires the assumption (Eq. 4), which would not be true for a general machine learning algorithms. Here, we verify this assumption for neural networks by numerical experiments. Specifically, we train a fully-connected neural networks using training datasets of different size $n$. We show that $\delta_{f}/\delta_{\mathcal{T}}$ is insensitive to training dataset size, and is always bounded from below by a positive constant $\kappa$ (Fig. 8). This result reveals that the neural networks would fit a dataset in a relatively smooth way during the training process.

5 Discussion

When neural networks are used to solve classification problems, we expect that the accuracy is dependent on some properties of the data set. It is still quite surprising, however, that there is a linear relationship between the accuracy and the cover complexity of the data set, as we have seen in Section 4.1. Theorem 3.20(ii) provides an upper bound of the error, but a lower bound is missing. To fully explain this observation, two conjectures of the learnability of fully-connected neural networks are proposed: when a neural network $f$ is trained on a data set $\mathcal{T}$ in such a way that $L_{f}^{max}(\mathcal{T})<\ln 2$ and $\delta_{f}(e^{-L}-c)\asymp\delta_{\mathcal{T}}$, then we have

$\mathcal{E}(f)\approx c(K)\cdot CC(\mathcal{T})$, where $c(K)$ is a constant depending only on $K$.

• 2.

$\frac{\mathcal{E}(f)}{\sqrt{K}}\approx c\cdot CC(\mathcal{T})$, where $c\approx 0.014$ is a constant.

On the other hand, the theoretical and numerical results provide a better understanding of the generalization of neural network from the training procedure. The smoothness $\delta_{f}(e^{-L}-c)$ of neural networks plays a key role, where $L$ is the maximum cross entropy loss $L_{f}^{max}$ or the average cross entropy loss $L_{f}$. We can see that:

$\delta_{f}(e^{-L}-c)$ depends on both the regularity of $f$ and the loss value $L$ (which also depends on $f$). Large $\delta_{f}(e^{-L}-c)$ requires good regularity and large $e^{-L}-c$, i.e., small $L$. However, small $L$ could correspond to bad regularity of $f$. Thus, there is a trade-off between the loss value $L$ and the regularity of $f$.

• 2.

Due to this trade-off, $\delta_{f}(e^{-L}-c)$ increases first and then decrease during the training process. Hence, we should not optimize neural networks excessively. Instead, we should stop the training early when $\delta_{f}(e^{-L}-c)$ begins to decrease, which leads to another “early stopping” strategy to prevent overfitting.

We also note that the lower bound of $\delta_{f}(e^{-L}-c)$ in Eq. (3) relates to the norm of weight matrices of neural networks:

[TABLE]

There have been some works to study the norm-based complexity of neural networks (see the Introduction), and these bounds typically scale with the product of the norms of the weight matrices, e.g., (Neyshabur et al., 2017a)

[TABLE]

where $h_{i}$ and $W_{i}$ are the number of nodes and the weight matrix in layer $i$ of a network with $l$-layers, and $\gamma_{margin}$ is the margin quantity, which describes the goodness of fit of the trained network to the data. The product of the matrix norms depends exponentially on the depth, while some recent works show that the generalization bound could scale polynomially in depth under some assumptions (Nagarajan and Kolter, 2019a; Wei and Ma, 2019). The exploration of the dependence of $\delta_{f}(e^{-L}-c)$ on depth is left for future work.

6 Conclusion

In this paper, we study the generalization error of neural networks for classification problems in terms of the data distribution and neural network smoothness. We first establish a new framework for classification problems. We introduce the cover complexity (CC) to measure the difficulty of learning a data set, an accuracy measure called $c$-accuracy which is stronger than the standard classification accuracy, and the inverse of the modulus of continuity to quantify neural network smoothness. Subsequently, we derive a quantitative bound for the expected accuracy/error in Theorem 3.20, which considers both the cover complexity and neural network smoothness.

We validate our theoretical results by several data sets of images. Our numerical results demonstrate that CC is a reliable measure for the difficulty of learning to classify a data set. On the other hand, we observe a clear consistency between test loss and neural network smoothness during the training process. We also show that neural network smoothness decreases when the network depth and width increases, and the effects of depth is more significant than that of width, while the smoothness is insensitive to training dataset size.

Acknowledgements

This work is supported by the DOE PhILMs project (No. de-sc0019453), the AFOSR grant FA9550-17-1-0013, and the DARPA AIRA grant HR00111990025. The work of P. Jin and Y. Tang is partially supported by the Major Project on New Generation of Artificial Intelligence from MOST of China (Grant No. 2018AAA0101002), and the National Natural Science Foundation of China (Grant No. 11771438).

Appendix A Example of topology

Example A.22**.**

Given

[TABLE]

[TABLE]

then $\tau_{T}$ is the topology generated by $U$.

In this example, $\{\{1\},\{1,2\},\{1,3\},\{1,2,3\}\}$ is an open set, since it consists of all elements containing label $\{1\}$, and $\{\{2,3\},\{1,2,3\}\}$ is also an open set with common part $\{2,3\}$. Besides open sets from base $U$, $\{\{1\},\{1,2\},\{1,3\},\{2,3\},\{1,2,3\}\}$ is still an open set as the union of the two shown above.

Appendix B Proof of Proposition 2.3

Proof.

We use the proof by contradiction. Assume that the result does not hold, then

$\exists\{x_{n}\},\{y_{n}\}\subseteq D,\ s.t.\ \|x_{n}-y_{n}\|\rightarrow 0(n\rightarrow\infty)$, and $tag(x_{n})\cap tag(y_{n})=\varnothing.$

Because $D$ is compact, there exist $x^{*}\in D$ and a subsequence $\{x_{k_{n}}\}$ of $\{x_{n}\}$ such that $\ x_{k_{n}}\rightarrow x^{*}.$ As $\|x_{n}-y_{n}\|\rightarrow 0$, also $\ y_{k_{n}}\rightarrow x^{*}.$ Choose any $i_{0}\in tag(x^{*})$, then there exists a sufficient large $k_{n_{0}}$ such that $\ i_{0}\in tag(x_{k_{n_{0}}}),\ i_{0}\in tag(y_{k_{n_{0}}}).$ Therefore $tag(x_{k_{n_{0}}})\cap tag(y_{k_{n_{0}}})\neq\varnothing$, which contradicts the assumption. ∎

Appendix C Proof of Proposition 2.4

Proof.

Let $\delta_{0}$ as defined in this proposition. For any two different points $x,y$ with distance less than $\delta_{0}$, either $tag(x)=tag(y)$, or at least one of the two is a full label point, in both cases $tag(x)\cap tag(y)\neq\varnothing$. For any $\delta>\delta_{0}$, according to the definition of $\delta_{0}$, there exist two points $x_{0},y_{0}$ satisfying

[TABLE]

The two facts imply that $\delta_{0}$ is the supremum of $\delta$ satisfying Proposition 2.3. ∎

Appendix D Proof of Proposition 2.7

Proof.

According to the definition,

[TABLE]

thus

[TABLE]

∎

Appendix E Estimate of total cover

In this section, we estimate the TC by the number of samples in the training set. The notations, such as $D$, $d$, $\mu$, $\rho_{\mathcal{T}}$, as well as training set

[TABLE]

are the same as before. Note that samples in $\mathcal{T}$ are drawn according to $\mu$. Before presenting the analysis, we first collect the following auxiliary notions and results (Definitions E.23-E.26, Theorem E.27) which are easily found in Mitzenmacher and Upfal (2017) (Definitions 14.1-14.3, Definition 14.5, and Theorem 14.8):

Definition E.23**.**

A range space is a pair $(X,\mathcal{R})$ where:

$X$* is a (finite or infinite) set of points;* 2. 2.

$\mathcal{R}$* is a family of subsets of $X$, called ranges.*

Definition E.24**.**

Let $(X,\mathcal{R})$ be a range space and let $Y\subseteq X$. The projection of $\mathcal{R}$ on $Y$ is

[TABLE]

Definition E.25**.**

Let $(X,\mathcal{R})$ be a range space. A set $Y\subseteq X$ is shattered by $\mathcal{R}$ if $|\mathcal{R}_{Y}|=2^{|Y|}$. The Vapnik-Chervonenkis (VC) dimension of a range space $(X,\mathcal{R})$ is the maximum cardinality of a set $Y\subseteq X$ that is shattered by $\mathcal{R}$. If there are arbitrarily large finite sets that are shattered by $\mathcal{R}$, then the VC dimension is infinite.

Definition E.26**.**

Let $(X,\mathcal{R})$ be a range space, and let $\mathcal{F}$ be a probability distribution on $X$. A set $N\subseteq X$ is an $\epsilon$-net for $X$ with respect to $\mathcal{F}$ if for any set $R\in\mathcal{R}$ such that $Pr_{\mathcal{F}}(R)\geq\epsilon$, the set $R$ contains at least one point from $N$, i.e.,

[TABLE]

Theorem E.27**.**

Let $(X,\mathcal{R})$ be a range space with VC dimension $d_{vc}$ and let $\mathcal{F}$ be a probability distribution on $X$. For any $0<\eta,\epsilon\leq 1/2$, there is an

[TABLE]

such that a random sample from $\mathcal{F}$ of size greater than or equal to $m$ is an $\epsilon$-net for $X$ with probability at least $1-\eta$.

Now let

[TABLE]

we first show $(D,B_{D})$ is a range space with VC dimension $d+1$.

Lemma E.28**.**

The VC dimension of range space $(D,B_{D})$ is $d+1$.

Proof.

The proof can be found in Dudley (1979). ∎

Set

[TABLE]

and

[TABLE]

we have the following lemmas.

Lemma E.29**.**

$\bigcup\limits_{A\in B_{D}^{\epsilon}(\frac{1}{2}r)}A\subseteq D\cap\bigcup\limits_{x_{i}\in\mathcal{T}}B(x_{i},r)$* when $\mathcal{T}$ is an $\epsilon$-net for $(D,B_{D})$.*

Proof.

For any $x\in\bigcup\limits_{A\in B_{D}^{\epsilon}(\frac{1}{2}r)}A$, we have $x\in A^{*}=B(x^{*},s^{*})\cap D$ for certain $s^{*}\leq\frac{1}{2}r$, with $\mu(A^{*})\geq\epsilon$. Since $\mathcal{T}$ is an $\epsilon$-net and $\mu(A^{*})\geq\epsilon$, we know $\mathcal{T}\cap A^{*}\neq\varnothing$. Thus there exists $x_{t}\in\mathcal{T}$ such that $\|x_{t}-x^{*}\|<s^{*}\leq\frac{1}{2}r$. Therefore

[TABLE]

The above inequality shows that

[TABLE]

∎

Lemma E.30**.**

$\lim\limits_{\epsilon\to 0}\varrho(\epsilon)=1$.

Proof.

For any positive decreasing sequence $\{\epsilon_{i}\}$ which satisfies $\lim_{i\to\infty}\epsilon_{i}=0$, it leads to an increasing chain

[TABLE]

where $\widetilde{D}=\bigcup_{A\in B_{D}^{+}(\frac{1}{2}r)}A$ for $B_{D}^{+}(\frac{1}{2}r)=\{A\in B_{D}|A=B(x,s)\cap D,0<s\leq\frac{1}{2}r,\mu(A)>0\}$. Let us consider a series of open balls $\{B_{i}\}$ of radius at most $\frac{1}{2}r$ that cover $D$, and we divide them into two parts $\{B_{i}\}=\{B_{i}^{+}\}\cup\{B_{i}^{0}\}$ such that $\mu(B_{i}^{+}\cap D)>0$ and $\mu(B_{i}^{0}\cap D)=0$. Then $\mu(\widetilde{D})\geq\mu(D\cap\bigcup_{i}B_{i}^{+})\geq\mu(D)-\mu(D\cap\bigcup_{i}B_{i}^{0})=1-0=1$, and thus $\mu(\widetilde{D})=1$. Therefore, we have $\lim_{\epsilon\rightarrow 0}\mu(\bigcup_{A\in B_{D}^{\epsilon}(\frac{1}{2}r)}A)=\mu(\widetilde{D})=1$.

Since

[TABLE]

by dominated convergence theorem, we have

[TABLE]

∎

According to the aforementioned lemmas, we deduce the following theorem.

Theorem E.31**.**

Let $\mathcal{T}=\{x_{1},x_{2},\dots,x_{n}\}$ be the training set drawn according to $\mu$, then for any $0<\eta,\epsilon\leq\frac{1}{2}$, there exists an

[TABLE]

such that

[TABLE]

holds with probability at least $1-\eta$ when $n\geq m$. Note that $\varrho(\epsilon)\rightarrow 1$ when $\epsilon\rightarrow 0$.

Proof.

Theorem E.27 shows that $\mathcal{T}$ is an $\epsilon$-net for range space $(D,B_{D})$ with probability at least $1-\eta$ when $n\geq m$. By lemma E.29, we have

[TABLE]

∎

From this theorem, we know that a large number of samples lead to a sufficiently large $\rho_{\mathcal{T}}$ with a high probability.

In the previous sections, there is an assumption that every training point has only one single correct label, so we will naturally consider this special case in the sequel.

Denote

[TABLE]

[TABLE]

[TABLE]

and $(D_{sin},B_{D_{sin}})$ is a range space with VC dimension at most $d+1$. Let

[TABLE]

that is, the samples in $\mathcal{T}$ are drawn according to $\mu_{sin}$. As before, denote

[TABLE]

[TABLE]

We have the following lemmas.

Lemma E.32**.**

$\bigcup\limits_{A\in B_{D_{sin}}^{\epsilon}(\frac{1}{2}r)}A\subseteq D\cap\bigcup\limits_{x_{i}\in\mathcal{T}}B(x_{i},r)$* when $\mathcal{T}$ is an $\epsilon$-net for $(D_{sin},B_{D_{sin}})$.*

Lemma E.33**.**

$\lim\limits_{\epsilon\to 0}\varrho_{sin}(\epsilon)=c_{sin}$, here

[TABLE]

[TABLE]

From these two lemmas we deduce the following theorem.

Theorem E.34**.**

Let $\mathcal{T}=\{x_{1},x_{2},\dots,x_{n}\}$ be the training set drawn according to $\mu_{sin}$, then for any $0<\eta,\epsilon\leq\frac{1}{2}$, there exists an

[TABLE]

such that

[TABLE]

holds with probability at least $1-\eta$ when $n\geq m$. Note that $\varrho_{sin}(\epsilon)\rightarrow c_{sin}$ when $\epsilon\rightarrow 0$ for $c_{sin}\geq\mu(D_{sin})$.

The proofs for Lemmas E.32-E.33 and Theorem E.34 are very similar to those for Lemmas E.29-E.30 and Theorem E.31, respectively. We omit them here. It is noteworthy that $c_{sin}$ is intuitively very close to 1, even equal to 1. At worst, $c_{sin}$ is at least greater than $\mu(D_{sin})$ which may be quite large in practice, and the proof is similar to what we show in the Lemma in E.30.

Appendix F Proof of Proposition 2.9

Proof.

Let $\mathcal{T}$ be the training set and $\lambda$ be a positive constant greater than 1, $\widetilde{\mathcal{T}}=\mathcal{T}/\lambda$, $\widetilde{\mu}(A)=\mu(\lambda A)$, then

[TABLE]

For the same reason,

[TABLE]

therefore

[TABLE]

∎

Appendix G Proof of Proposition 3.17

Proof.

Denote by $\widetilde{\mathcal{T}}_{c_{1}}=\{x_{i}\in\mathcal{T}\ |\ f$ is $c_{1}-$accurate at x_{i}$$\}, $\delta=\min(\delta_{0},\delta_{f}(c_{1}-c_{2}))$. For any $x_{i}\in\widetilde{\mathcal{T}}_{c_{1}}$, choose $k_{i}\in\{1,\cdots,K\}$ such that $tag(x_{i})=\{k_{i}\}$. From the definition of $\widetilde{\mathcal{T}}_{c_{1}}$ we know that $f_{k_{i}}(x_{i})>c_{1}$.

For any $x\in B(x_{i},\delta)\cap D$, from Proposition 2.3 we know that $tag(x)\cap tag(x_{i})\neq\varnothing$, and hence $k_{i}\in tag(x)$. On the other hand, $\|f(x)-f(x_{i})\|_{\infty}<c_{1}-c_{2}$, so we have $|f_{k_{i}}(x)-f_{k_{i}}(x_{i})|<c_{1}-c_{2}$, therefore

[TABLE]

which means that $f$ is $c_{2}-$accurate at $x$, that is to say

[TABLE]

Then

[TABLE]

The second inequality can be derived from Proposition 2.7. ∎

Appendix H Proof of Theorem 3.18

Proof.

Define $\delta:=\min(\delta_{0},\delta_{f}(e^{-L_{f}^{max}}-c))$. Note that if $x_{i}\in\mathcal{T}$, and $x\in B(x_{i},\delta)\cap D$, then $\|x-x_{i}\|<\delta_{0}$, and hence $tag(x)\cap tag(x_{i})\neq\varnothing$, so that $k_{i}\in tag(x)$.

Because of

[TABLE]

we have

[TABLE]

so that $|f_{k_{i}}(x)-f_{k_{i}}(x_{i})|<e^{-L_{f}^{max}}-c$. Therefore

[TABLE]

so that $f$ is c-accurate at $x$. Overall we obtain

[TABLE]

The second inequality can be derived from Proposition 2.7. ∎

Appendix I Proof of Theorem 3.19

Proof.

We will prove it by contradiction. Consider two points $x_{1}$ and $x_{2}$ with different labels, and $\|x_{1}-x_{2}\|=\delta_{\mathcal{T}}$.

If $\delta_{f}(e^{-L_{f}^{max}}-c)>\delta_{\mathcal{T}}/2$, then by the definition of $\delta_{f}(e^{-L_{f}^{max}}-c)$, $\|f(\frac{x_{1}+x_{2}}{2})-f(x_{1})\|_{\infty}<e^{-L_{f}^{max}}-c$, since $\|\frac{x_{1}+x_{2}}{2}-x_{1}\|=\|\frac{x_{2}-x_{1}}{2}\|=\delta_{\mathcal{T}}/2$. Similarly, $\|f(\frac{x_{1}+x_{2}}{2})-f(x_{2})\|_{\infty}<e^{-L_{f}^{max}}-c$. Then we have

[TABLE]

On the other hand, by the definition of $L_{f}^{max}$, $-\ln f_{k_{1}}(x_{1})\leq L_{f}^{max}$ and $-\ln f_{k_{2}}(x_{2})\leq L_{f}^{max}$, so $f_{k_{1}}(x_{1})\geq e^{-L_{f}^{max}}$, $f_{k_{2}}(x_{2})\geq e^{-L_{f}^{max}}$ and $f_{k_{1}}(x_{2})\leq 1-f_{k_{2}}(x_{2})\leq 1-e^{-L_{f}^{max}}$, therefore

[TABLE]

∎

Appendix J Proof of Theorem 3.20

Proof.

From Theorem 3.18 and assumption we know that

[TABLE]

which implies $\lim\limits_{\rho_{\mathcal{T}}\to 1}p(f)=1$. Note that $\kappa$ is less than $0.5$ and $\delta_{0}$ is only determined by the classification problem itself. The above inequality is easy to convert into form (ii). ∎

Appendix K Detailed information of data and parameters for training

First, we list the information concerning the data selection.

MNIST: Last 55000 samples of the training set for training and all the 10000 samples of the test set for testing. 2. 2.

CIFAR-10: First 49000 samples of the training set for training and all the 10000 samples of the test set for testing. 3. 3.

CIFAR-100: First 49000 samples of the training set for training and all the 10000 samples of the test set for testing. 4. 4.

COIL-20: 1200 samples whose end numbers of the figure names are not multiples of 6 for training and 240 samples whose end numbers are multiples of 6 for testing. 5. 5.

COIL-100: 6000 samples whose end numbers of the figure names are not multiples of 30 for training and 1200 samples whose end numbers are multiples of 30 for testing. 6. 6.

SVHN: First 50000 samples of the training set for training and first 10000 samples of the test set for testing.

Parameters for networks are listed in Table 3.

For generating convolution data, we choose the following structure

[TABLE]

with kernel size $3\times 3$(strides $1$) and pool size $2\times 2$(strides $2$), then train this CNN with batch size $64$, learning rate $0.001$ and optimizer RMSProp for 5 epochs. After that, extract new data at location mentioned in above structure by feeding the data to the trained network.