Generalization from correlated sets of patterns in the perceptron

TL;DR

This paper introduces a framework to quantify how easily a neural network can unify interpretations of different pattern sets based on their dissimilarity, using the perceptron as a case study.

Contribution

It proposes a novel distance-based measure of generalization and analytically computes the capacity of the perceptron to unify pattern sets at various dissimilarities.

Findings

Generalization capacity decreases with pattern dissimilarity.

Analytical expression for the perceptron's distance-based capacity.

Generalization remains possible at any dissimilarity, but with reduced capacity.

Abstract

Generalization is a central aspect of learning theory. Here, we propose a framework that explores an auxiliary task-dependent notion of generalization, and attempts to quantitatively answer the following question: given two sets of patterns with a given degree of dissimilarity, how easily will a network be able to "unify" their interpretation? This is quantified by the volume of the configurations of synaptic weights that classify the two sets in a similar manner. To show the applicability of our idea in a concrete setting, we compute this quantity for the perceptron, a simple binary classifier, using the classical statistical physics approach in the replica-symmetric ansatz. In this case, we show how an analytical expression measures the "distance-based capacity", the maximum load of patterns sustainable by the network, at fixed dissimilarity between patterns and fixed allowed number of errors. This curve indicates that generalization is possible at any distance, but with decreasing capacity. We propose that a distance-based definition of generalization may be useful in numerical experiments with real-world neural networks, and to explore computationally sub-dominant sets of synaptic solutions.

Figures

Equations

$$d_{\text{H}}({\mathbf{\xi }}_{\mu },{\mathbf{\xi }}_{\nu })=\frac{1}{2}-\frac{1}{2N}{\mathbf{\xi }}_{\mu }\cdot {\mathbf{\xi }}_{\nu }.$$

$$d_{\text{in}}(\xi ,\bar{\xi })=\underset{\pi \in S_p}{\min }\sum _{\mu }d({\mathbf{\xi }}_{\mu },{\bar{\mathbf{\xi }}}_{\pi (\mu )}),$$

$$P_{d_{\text{in}}}(\xi ,\bar{\xi }):$$

$$=\frac{1}{P(d_{\text{in}}{)}^p}\prod _{\mu =1}^{p}P({\mathbf{\xi }}_{\mu })P({\bar{\mathbf{\xi }}}_{\mu })\delta (d_{\text{H}}({\mathbf{\xi }}_{\mu },{\bar{\mathbf{\xi }}}_{\mu })-d_{\text{in}})$$

$$d_{\text{out}}(\sigma ,\bar{\sigma })=\frac{1}{p}\sum _{\mu =1}^p(1-{\delta }_{{\sigma }_{\mu },{\bar{\sigma }}_{\mu }}).$$

$$H_{(\xi ,\sigma )\cup (\bar{\xi },\bar{\sigma })}(W)=H_{\xi ,\sigma }(W)+H_{\bar{\xi },\bar{\sigma }}(W),$$

$$Z(\beta ,(\xi ,\sigma ))=\int dW{e}^{-\beta H_{\xi ,\sigma }(W)},$$

$$F(\beta ,(\xi ,\sigma ))=-\frac{1}{\beta N}\ln Z(\beta ,(\xi ,\sigma ))$$

$${\Omega }_{\text{n}}((\xi ,\sigma ),(\bar{\xi },\bar{\sigma }),\beta ):=\frac{1}{\beta N}\ln \frac{\sqrt{Z(\beta ,(\xi ,\sigma ))Z(\beta ,(\bar{\xi },\bar{\sigma }))}}{Z(\beta ,(\xi ,\sigma )\cup (\bar{\xi },\bar{\sigma }))}.$$

$${\Omega }_{\text{n}}((\xi ,\sigma ),(\bar{\xi },\bar{\sigma }),\beta )=-\frac{F(\xi ,\sigma )+F(\bar{\xi },\bar{\sigma })}{2}+F((\xi ,\sigma )\cup (\bar{\xi },\bar{\sigma }))$$

$$\Omega (D_{\text{out}},d_{\text{in}})={⟨{\Omega }_{\text{n}}((\xi ,\sigma ),(\bar{\xi },\bar{\sigma }),\beta )⟩}_{d_{\text{in}}(\xi ,\bar{\xi })=d_{\text{in}};d_{\text{out}}(\sigma ,\bar{\sigma })=D_{\text{out}}}.$$

$$H_{\xi ,\sigma }(\mathbf{W})=\sum _{\mu =1}^p\Theta (-{\sigma }_{\mu }{\mathbf{\xi }}_{\mu }\cdot \mathbf{W}).$$

$$\begin{array}{rl}dP(\mathbf{W})& =\frac{d\mathbf{W}\delta \left({\sum }_{i=1}^N{W}_i^2-N\right)}{{\int }_{{\mathbb{R}}^N}d\mathbf{W}\delta \left({\sum }_{i=1}^N{W}_i^2-N\right)}(\mathrm{spherical})\\ dP(\mathbf{W})& =\frac{1}{2^N}(\mathrm{discrete}).\end{array}$$

$${\alpha }_c(\beta )=\underset{\alpha }{\inf }\{\alpha :{⟨H⟩}_{(\xi ,\sigma )}>0\}$$

$$P(\xi ,\sigma )=\prod _{\mu =1}^p[a{\delta }_{{\sigma }_{\mu },-1}+(1-a){\delta }_{{\sigma }_{\mu },1}]\prod _{j=1}^N[b{\delta }_{{\xi }_{\mu }^j,-1}+(1-b){\delta }_{{\xi }_{\mu }^j,1}].$$

$$P(\xi ,\sigma )=P(\xi )\prod _{\mu =1}^p\delta ({\sigma }_{\mu }-{\sigma }_T({\xi }_{\mu }))$$

$$F(D_{\mathrm{out}})=\frac{1}{N}$$

$$\displaystyle\sum_{\{\epsilon_{\mu}=\pm 1\}}\delta\Big{(}\sum_{\mu=1}^{p}\epsilon_{\mu}-p(1-2D_{\textrm{out}})\Big{)}\;\prod_{\mu=1}^{p}\Theta(\boldsymbol{W}\cdot\boldsymbol{\xi}_{\mu})\;\Theta(\epsilon_{\mu}\boldsymbol{W}\cdot\bar{\boldsymbol{\xi}}_{\bar{\mu}}).$$

$$\sum_{\{\epsilon_{\mu}=\pm 1\}}\delta\Big{(}\sum_{\mu=1}^{p}\epsilon_{\mu}-p(1-2D_{\textrm{out}})\Big{)}\;\prod_{\mu=1}^{p}\Theta(\boldsymbol{W}\cdot\boldsymbol{\xi}_{\mu})\;\Theta(\epsilon_{\mu}\boldsymbol{W}\cdot\bar{\boldsymbol{\xi}}_{\bar{\mu}})$$

$$\Omega (D_{\text{out}}|d_{\text{in}})=\overline{F_{\alpha }}-\overline{F_{\alpha }(D_{\text{out}}|d_{\text{in}})}$$

$$\overline{F_{\alpha }(D_{\text{out}}|d_{\text{in}})}=\int P_{d_{\text{in}}}(\xi ,\bar{\xi })F_{\alpha }(\xi ,\bar{\xi },D_{\text{out}}).$$

$${\alpha }_c(D_{\text{out}}|d_{\text{in}})=\underset{\alpha }{\inf }\{\alpha :\overline{F_{\alpha }(D_{\text{out}}|d_{\text{in}})}>-\infty \}.$$

$$P(D_{\text{out}}|d_{\text{in}})$$

$$=\exp [NF(D_{\text{out}}|d_{\text{in}})-N\underset{D^{\prime }}{\max }F(D^{\prime }|d_{\text{in}})]\to \{\begin{array}{cc}1& \text{mbox}finitechance\\ 0& \text{mbox}nochance\end{array}$$

$$\overline{F_{\alpha }(D_{\text{out}}|d_{\text{in}})}=\int P(\xi )\ln Z=\underset{m\to 0}{\lim }\frac{1}{m}\ln \int P(\xi )Z^m$$

$$1=\int \prod _{a<b}d{Q}_{ab}\delta (Q_{ab}-{\mathbf{W}}_a\cdot {\mathbf{W}}_b/N).$$

$$\int P(\xi )Z^m=\int P(\xi )\int \prod _{a=1}^{m}d{\mathbf{W}}_a{e}^{-\beta H_{\xi }({\mathbf{W}}_a)}=:\int \prod _{a<b}d{Q}_{ab}{e}^{NA[\{Q_{ab}\}]}.$$

$$\begin{gathered} \overline{F_{\alpha }(D_{\text{out}}|d_{\text{in}})}=\underset{m\to 0}{\lim }\frac{1}{Nm} \\ \ln \int dQd\hat{Q}{e}^{Nm[G_0(\hat{Q})+G_1(Q,d_{\text{in}},D_{\text{out}})-iQ\hat{Q}/2]}. \end{gathered}$$

$$G_0^{\text{spherical}}(\hat{Q})=-\sqrt{i\hat{Q}}-i\hat{Q}/2,$$

$$G_0^{\text{discrete}}(\hat{Q})=\ln 2-i\hat{Q}/2+\frac{1}{\sqrt{2\pi }}\int dx\exp \left(-\frac{1}{2}{x}^2\right)\ln \cosh \left[x\sqrt{i\hat{Q}}\right].$$

$$\begin{array}{rl}{G}_1(& Q,d_{\text{in}},D_{\text{out}})=\\ & (1-D_{\text{out}})\left[\int {\mathcal{D}}_{d_{\text{in}}}(y,\bar{y})\ln {\int }_{x>y;\bar{x}>\bar{y}}{\mathcal{D}}_{d_{\text{in}}}\left(\sqrt{\frac{Q}{1-Q}}x,\sqrt{\frac{Q}{1-Q}}\bar{x}\right)\right]\\ +& D_{\text{out}}\left[\int {\mathcal{D}}_{d_{\text{in}}}(y,\bar{y})\ln {\int }_{x>y;\bar{x}<\bar{y}}{\mathcal{D}}_{d_{\text{in}}}\left(\sqrt{\frac{Q}{1-Q}}x,\sqrt{\frac{Q}{1-Q}}\bar{x}\right)\right]\\ +& D_{\text{out}}\ln D_{\text{out}}+(1-D_{\text{out}})\ln (1-D_{\text{out}}),\end{array}$$

$$\mathcal{D}_{d_{\textrm{in}}}(x,\bar{x})=dx\;d\bar{x}\frac{1}{2\pi}\exp\Bigg{(}-\frac{1}{2\sqrt{1-q^{2}}}\begin{bmatrix}x&\bar{x}\end{bmatrix}\begin{bmatrix}1&-q\\ -q&1\end{bmatrix}\begin{bmatrix}x\\ \bar{x}\end{bmatrix}\Bigg{)},$$

Full text

1 Introduction

Generalization is an essential feature of cognition. It constructs broad, universal statements or general concepts from a few empirical observations. Our ability to generalize comprises a wide and not well characterized set of tasks and abilities, including the tendency to respond in the same way to different, but similar “on some level”, stimuli. An important feature of generalization is to assume or recognize the existence of common features shared by sets of elements. Hence, classifying common relations and differences among different observed patterns is crucial [1].

The problem of breaking down the process of generalization into its elementary components naturally emerges in the field of artificial neural networks. In this context, achieving a deeper understanding of generalization could improve the current design principles and clarify the reasons why current architectures generalize well. A generally held belief is that the efficiency of deep neural networks in identifying patterns can be understood in terms of feature extraction. Despite its robustness, this view is being challenged by an increasing number of experimental results. For example, a recent study showed that this paradigm is misleading when trying to understand how deep neural networks generalize, and proposes to consider alternative approaches [2]; one such approach already emerged in a statistical mechanics setting [3, 4]. Another challenging observation is a recently discovered fragility of neural networks to so-called “adversarial attacks” [5], whereby any given pattern can be maliciously modified into a very similar pattern that gets misclassified.

Learning efficiently from examples requires a way to gauge the size and quality of the training and test sets. The classic methods to address these issues (e.g., bias-variance decomposition, Vapnik-Chervonenkis dimension, model complexity) are found within so-called “statistical learning theory” [6]. In this framework, the goal of learning is the approximation of a function $f$ by an element $h$ of a given hypothesis space $\mathcal{H}$. The trained machine $h$ is chosen by a learning algorithm, starting from a training set, i.e. a set of pairs $\xi_{\mu},f(\xi_{\mu})$, where $\xi_{\mu}$ are chosen from a space $\Omega$ with unknown probability distribution. Statistical learning theory establishes general relations between the complexity or expressivity of a given function class $\mathcal{H}$ and its generalization properties, defined as the ability of its elements to reproduce the output of $f$ beyond the training set [7]. Despite its power, this framework has limitations concerning its applicability to neural networks [8]. For instance, it was observed recently that over-parameterized neural networks often generalize better than smaller, less complex, ones. This finding is in conflict with the predictions of statistical learning and classical computational complexity theory (as well as with naive intuition) [9].

Another important drawback of statistical learning theory is that it considers generalization in a worst-case scenario, where the capabilities of the network are tested against possibly malicious counterexamples [10]. A statistical physics approach overcomes this problem by considering the so-called “teacher-student” scenario [4, 11, 12]. This framework usually assumes that the generalization ability of a trained network (the student) is measured on a set of tests produced by a teacher with identical architecture. An important limitation of the teacher-student setting is that the student can learn from irrelevant examples if the teacher is used as a surrogate for a classification rule not involving the whole space of inputs. Consider, for instance, a teacher trained to discern handwritten digits from a training set $\Omega_{\mathrm{t}}$ (e.g. from the popular MNIST database). The training set is a subset of a much larger set $\Omega$ of all black-and-white images. The teacher will associate well-defined output values to elements of $\Omega_{\mathrm{t}}$ but also to any other element of $\Omega$, for instance to those in a set $\Omega_{\mathrm{n}}$, disjoint from $\Omega_{\mathrm{t}}$, containing random noise. Now the student can in principle learn to reproduce the teacher’s response on $\Omega_{\mathrm{t}}$ by learning to mimic the teacher on the (meaningless) set $\Omega_{\mathrm{n}}$. In other words, the student can learn to classify a dataset without ever seeing a single example from it. This problem is connected to the more general fact that the space $\Omega$, in practical applications, has structure, first and foremost via notions of distance or similarity between its elements.

Building on these premises, we propose here an auxiliary way to approach generalization. The main object of interest is the number of different synaptic configurations that allow the network to classify correctly two sets of patterns. This quantity is a generalization of the classic “Gardner volume” [13, 14] to the case where the training set consists of two sets of patterns with a prescribed degree of correlation. The rationale is to take explicitly into account the degree of similarity between “new” and “old” information seen by a network. Here we use a simple definition of distance between binary data sets, based on their overlaps, but the framework applies to any other definition of distance. The load at which the Gardner volume becomes zero defines a capacity, which in our case depends on the distance. To root our ideas in a concrete case, we carry out these calculations on a very simple neural network, the perceptron, using classic results from statistical mechanics [13, 15].

2 Definition of the approach and results

2.1 Measuring the similarity between classification tasks

This section defines two complementary notions of dissimilarity between classification tasks, one based on a bit-wise comparison of the training sets, and the other based on the free-energy landscape of the corresponding supervised learning problem.

A classifier neural network is described by an output function $\sigma$ that associates an output $\sigma(\boldsymbol{\xi}_{\mu})$ to any input vector $\boldsymbol{\xi}_{\mu}\in\mathbb{R}^{N}$. We consider binary input vectors, whose elements are $\pm 1$ bits and we define a classification task as a training set $\xi=\{\boldsymbol{\xi}_{\mu}\}_{\mu=1,...,p}$ with associated labels $\sigma=\{\sigma_{\mu}\}_{\mu=1,...,p}$. Therefore, a task is a set of input-output pairs $(\xi,\sigma)=\{(\boldsymbol{\xi}_{\mu},\sigma_{\mu})\}_{\mu=1,..,.p}$. The network can solve the task if $\forall\mu=1,...,p$, $\sigma_{\mu}=\sigma(\boldsymbol{\xi}_{\mu})$.

Let us fix two classification tasks, say $(\xi,\sigma)$ and $(\bar{\xi},\bar{\sigma})$. We focus on inputs ($\xi$ and $\bar{\xi}$) and outputs ($\sigma$ and $\bar{\sigma}$) separately, thus defining two bit-wise distances. The canonical distance between two patterns $\boldsymbol{\xi}_{\mu}$ and $\boldsymbol{\xi}_{\nu}$ is the normalized Hamming distance, defined as

[TABLE]

The quantity $\boldsymbol{\xi}_{\mu}\cdot\boldsymbol{\xi}_{\nu}/N=\sum_{i=1}^{N}\xi_{\mu}^{i}\xi_{\nu}^{i}/N=:q(\boldsymbol{\xi}_{\mu},\boldsymbol{\xi}_{\nu})$ is the overlap between $\boldsymbol{\xi}_{\mu}$ and $\boldsymbol{\xi}_{\nu}$. We need to extend this definition to the full sets of inputs $\xi=\{\boldsymbol{\xi}_{\mu}\}_{\mu=1,\ldots,p}$ and $\bar{\xi}=\{\bar{\boldsymbol{\xi}}_{\bar{\mu}}\}_{\bar{\mu}=1,\ldots,p}$. Intuitively, $\xi$ and $\bar{\xi}$ are identical if, for any $\boldsymbol{\xi}_{\mu}\in\xi$ there exists some $\bar{\boldsymbol{\xi}}_{\bar{\mu}}\in\bar{\xi}$ such that $\boldsymbol{\xi}_{\mu}=\bar{\boldsymbol{\xi}}_{\bar{\mu}}$. Following this line of reasoning, one can define a distance between the sets $\xi$ and $\bar{\xi}$ as

[TABLE]

where $S_{p}$ is the symmetric group of order $p$. The minimum over $S_{p}$ selects the closest matching between each pattern in $\xi$ and each one in $\bar{\xi}$, thus restoring the symmetry under permutations, but it complicates the analytical computations. Therefore, we adopt a simpler setup, still inspired by (2), where the elements of the input sets are at fixed distance $d$ pairwise, meaning that $d_{\textrm{in}}(\xi_{\mu},\bar{\xi}_{\mu})=d_{\textrm{in}}$ for every $\mu$. We consider the ensemble of $p$ independent pairs of inputs at Hamming distance $d_{\textrm{in}}$, defined by the joint probability

[TABLE]

where $P(d_{\textrm{in}}):=P(d_{\textrm{H}}(\boldsymbol{\xi}_{1},\boldsymbol{\xi}_{2})=d_{\textrm{in}})$ is the so-called Bayesian normalization, i.e., the probability that a pair of two random vectors $\boldsymbol{\xi}_{1}$ and $\boldsymbol{\xi}_{2}$ has Hamming distance $d_{\textrm{in}}$.

Having fixed the matching between inputs by this definition, it is then natural to define the distance between outputs $\sigma=\{\sigma_{\mu}\}_{\mu=1,...,p}$ and $\bar{\sigma}=\{\bar{\sigma}_{\bar{\mu}}\}_{\bar{\mu}=1,...,p}$ as

[TABLE]

We now focus on the similarity between two tasks in terms of synaptic representation. We reason that, from the point of view of the network, two tasks are similar if they are easily solved by the same synaptic configuration, i.e., if they share a significant fraction of common solutions. Here, by solutions we mean the solutions to the problem of finding a state of the network, specified by a synaptic weight structure $W$, such that the task is solved correctly. Two equivalent tasks will share all solutions and should have zero synaptic distance. Conversely, two tasks are incompatible if they have no common solution, or, equivalently, if there exists no solution to the task of learning the union of the two tasks. In this section, we do not specify the nature of $W$ any further, since our approach can be applied to more general multi-layered architectures. In the following sections we apply our ideas to the case of the perceptron only.

A definition of synaptic distance consistent with these intuitive requirements can be formalized as follows. Let us consider a neural network with cost function $H_{\xi,\sigma}(W)$, equal, for instance, to the number of errors that the network makes in performing the task $(\xi,\sigma)$ when its synaptic weights are $W$. Let the cost function be additive under union of the training sets: given two sets of input-output pairs $(\xi,\sigma)$ and $(\bar{\xi},\bar{\sigma})$ of arbitrary sizes $p$ and $\bar{p}$,

[TABLE]

where $(\xi,\sigma)\cup(\bar{\xi},\bar{\sigma})$ denotes the labelled training set with inputs $\left\{\boldsymbol{\xi}_{1},\ldots,\boldsymbol{\xi}_{p},\bar{\boldsymbol{\xi}}_{1},\ldots,\bar{\boldsymbol{\xi}}_{\bar{p}}\right\}$ and outputs $\left\{\sigma_{1},\ldots,\sigma_{p},\bar{\sigma}_{1},\ldots,\bar{\sigma}_{\bar{p}}\right\}$. The canonical partition function at inverse temperature $\beta$ of a system with Hamiltonian $H_{\xi,\sigma}$ is

[TABLE]

where $dW$ is a normalized measure on the synaptic weights. In the zero-temperature limit $\beta\to\infty$, the system occupies the lowest-energy state and $Z$ becomes the degeneracy of the ground state, or the fraction of exact solutions to the task $(\xi,\sigma)$. The free energy is, in this context, referred to as the Gardner volume:

[TABLE]

Our definition of network distance is then

[TABLE]

In the zero-temperature limit $\beta\to\infty$ this quantity counts the number of exact common solutions to the two tasks, normalized by the number of solutions to the two tasks separately. In fact, Eq. (9) can be rewritten in terms of the free energy as

[TABLE]

The synaptic distance is zero for two identical tasks and diverges for incompatible tasks: $\Omega_{\textrm{n}}((\xi,\sigma),(\xi,\sigma),\infty)=0$ and $\Omega_{\textrm{n}}((\xi,\sigma),(\xi,-\sigma),\infty)=\infty$.

In this setting, the ability of the network to generalize can be studied by comparing the two types of distances defined above. In particular, we will consider the typical value of $\Omega$ in the ensemble where $d_{\textrm{in}}=d_{\textrm{in}}(\xi,\bar{\xi})$ and $D_{\textrm{out}}=d_{\textrm{out}}(\sigma,\bar{\sigma})$ are fixed:

[TABLE]

A critical line is identified by the point in which $\Omega=\infty$ for fixed $(D_{\textrm{out}},d_{\textrm{in}})$ for a given size of the input sets. Equivalently, for fixed $d_{\textrm{in}}$ and size $p$, $\Omega=\infty$ indicates the theshold values of $D_{\textrm{out}}$, i.e. the range of output-similarity that the network can typically attribute to the input sets.

2.2 Generalization properties of the perceptron

We now set out to specify the abstract notion of memory-based distance introduced in the previous section (Eq. (11)) in order to use it for an explicit calculation. We call $\xi=\{\boldsymbol{\xi}_{\mu}\}_{\mu=1,...,p}$ a set of $p=\alpha N$ input vectors with components $\xi_{\mu}^{i}=\pm 1\;\;\forall i=1,...,N$. The perceptron is a network which yields a binary output $\sigma_{W}(\xi_{\mu})=\operatorname{sign}(\boldsymbol{W}\cdot\boldsymbol{\xi}_{\mu}/N-K)=\pm 1$ for any input, given a certain synaptic configuration $\boldsymbol{W}\in\mathbb{R}^{N}$. In the spherical model $\sum_{i}W_{i}^{2}=N$, in the discrete model $W_{i}=\pm 1$.

In the case of batch learning, for any given training set $(\xi,\sigma)$, the energy conventionally associated to this model is the error-counting cost function

[TABLE]

This definition allows to compute the Gardner volume, i.e. the fraction of synaptic configurations $\boldsymbol{W}$ that solve a certain task, as defined in (8).

As for the probabily of synaptic configurations $dP(\boldsymbol{W})$, the following maximally entropic probability distributions are conventionally used, for the spherical and discrete case respectively

[TABLE]

A different cost function was proposed recently, in order to study a peculiar clustering property of the solutions in the perceptron with discrete weigths [16, 17, 18].

In order to study the typical behaviour of this network, the Gardner volume should be averaged on the input-output pair statistics $P(\xi,\sigma)$ (quenched average). A second-order phase transition is witnessed by the average value of the cost function. The critical capacity $\alpha_{c}(\beta)$ is defined as

[TABLE]

and identifies a critical line in the $(\alpha,\beta)$ plane. Physically, $\alpha_{c}(\beta)$ is the maximum number of patterns per neuron that can be learned in the typical case (without committing an extensive number of mistakes). The value of $\alpha_{c}$ clearly depends on the statistics of the input-ouput pairs. The simplest statistics is obtained by choosing both inputs and outputs randomly and independently

[TABLE]

In the unbiased case, $a=b=1/2$ and $\beta=\infty$, $\alpha_{c}=2$ for the spherical perceptron, while $\alpha_{c}\approx 0.833$ in the discrete case. It is important to point out that, in the latter case, the replica-symmetric ansatz yields a quantitatively incorrect result and one-step replica symmetry breaking is needed [19].

In the case of a single set of independent and spatially correlated inputs, two classic studies [20, 21], derive the capacity from the (intra) correlation matrix $C_{ij}=\langle\xi_{\mu}^{i}\xi_{\mu}^{j}\rangle$. More recent studies [22, 23] have focused on the capacity in the case of a prescribed correlation between different patterns, as given by the overlap matrix $C_{\mu\nu}=\boldsymbol{\xi}_{\mu}\cdot\boldsymbol{\xi}_{\nu}/N$.

The teacher-student setting for generalization assumes the following specific choice of the output statistics. Instead of drawing inputs and outputs independently, there is an input (example) statistics $P(\xi)$ and a teacher machine, described by an output function $\sigma_{T}$. For each input, the “correct” output is chosen by the teacher. Specifically, the input-output statistics is given by

[TABLE]

in a noise-free scenario. In this scenario, the cost function (called “learning function”), is algorithm-specific, and its average value $\langle H\rangle$, as a function of the number of examples $\alpha=p/N$, is called “learning curve” $\epsilon(\alpha)$. The learning curve quantifies generalization. The generalization ability can be defined by averaging on all possible teachers, if needed. For a perceptron, the best possible performance is achieved with the Bayesian algorithm which, however, can only be performed by a perceptron-based committee machine [24, 25, 15].

2.2.1 Distance-based Gardner volume

Turning to our approach to define generalization, we choose both inputs $\xi$ and outputs $\sigma$ to be random and unbiased. Since we have two sets $(\xi,\sigma)$ and $(\bar{\xi},\bar{\sigma})$, the standard gauge freedom of the problem allows to fix the outputs of the first set $\sigma_{\mu}$ to $+1$. With this premise, we write the non-trivial part of Eq. (10), for given $d_{\mathrm{out}}(\sigma,\bar{\sigma})=D_{\mathrm{out}}$, as

[TABLE]

Note that this is a “zero-temperature” definition, and that

[TABLE]

equals one if exactly $D_{\textrm{out}}p$ output pairs are discordant, regardless of which pairs, and zero otherwise. This choice of summing over $\epsilon_{\mu}$ inside the logarithm makes Eq. (17) slightly different from Eq. (10). Nonetheless, the two quantities are equal up to a combinatorial prefactor which is irrelevant for the final result. This also justifies the conditional notation $(D_{\textrm{out}}|d_{\textrm{in}})$ once the average over input sets of given distance is taken.

The full expression for (11) is therefore

[TABLE]

where $\overline{F_{\alpha}}=\overline{F_{\alpha}(0|0)}$ is the conventional Gardner volume for $N\alpha$ inputs, and

[TABLE]

Both these observables can be evaluated within the replica formalism, as will be outlined in the next section (a detailed derivation will be given in Appendix A and B).

Finally, we introduce the distance-based capacity $\alpha_{c}(D_{\textrm{out}}|d_{\textrm{in}})$, which is to $\overline{F_{\alpha}(D_{\textrm{out}}|d_{\textrm{in}})}$ what $\alpha_{c}$ (Eq. (14)) is to the Gardner volume (7). Since in our “zero temperature” framework we have not explicitly introduced a cost function, it is convenient to define $\alpha_{c}(D_{\textrm{out}}|d_{\textrm{in}})$ as

[TABLE]

Physically, $\alpha_{c}(D_{\textrm{out}}|d_{\textrm{in}})N$ is the maximum size of two sets, with distance $d_{\textrm{in}}$, that can be learned simultaneously with $D_{\textrm{out}}\;\alpha_{c}(D_{\textrm{out}}|d_{\textrm{in}})N$ concordant output pairs.

Now, suppose the perceptron has learned a set $\xi$ of size $\alpha N$. If a new set $\bar{\xi}$, with $d_{\textrm{in}}(\xi,\bar{\xi})=d_{\textrm{in}}$ is presented, then there will be a fraction $D_{\textrm{out}}$ of discordant outputs and in this case $\alpha_{c}(D_{\textrm{out}}|d_{\textrm{in}})$ shows the range $D_{\textrm{out}}$ can assume, given $d_{\textrm{in}}$. It must be remarked that this boundary does not show whether any $D_{\textrm{out}}$ has a finite probability. A related quantity is the conditional probability $P(D_{\textrm{out}}|d_{\textrm{in}})$, which may be approximated by

[TABLE]

However, this idea requires some non-trivial computations and careful geometrical considerations, and we do not pursue it here.

2.2.2 Analytical expression for the distance-based capacity

We are able to derive an analytical expression for the critical capacity at fixed pattern distance, which we discuss in the following. The quantity $\overline{F_{\alpha}(D_{\textrm{out}}|d_{\textrm{in}})}$ in Eq. (20) is, formally, an average free energy and can be computed with the replica formalism [13, 15]. The replica method is based on the identity

[TABLE]

Following standard procedure, we introduce the parameter $Q_{ab}$ by

[TABLE]

Hence

[TABLE]

As $N\to\infty$ we compute the stability equations for $A[Q]$ and obtain $Q$. It is known that the so called replica-symmetry (RS) ansatz $Q_{ab}=(1-Q)\delta_{ab}+Q$ leads to the correct result for the spherical perceptron, while it is not correct in the discrete case. Hence, the limit $Q\to 1^{-}$ identifies the point in which all the solutions collapse onto a single one (up to subextensive contributions) and it yields the critical point $\alpha_{c}$ [15].

In our model, this procedure (outlined in Appendix A) yields

[TABLE]

The function $G_{0}$ depends on the model. For the spherical perceptron

[TABLE]

while for the discrete model

[TABLE]

The second function is common to the two models:

[TABLE]

where

[TABLE]

with $q=1-2d_{\textrm{in}}$. It is worth noticing that in the limits $d_{\textrm{in}}\to 0$ and $D_{\textrm{out}}\to 0$, $G_{0}$ and $G_{1}$ reduce to Gardner and Derrida’s RS formulas [13]. Finally, the saddle point equations are

[TABLE]

and are studied in Appendix B.

In order to compute the generalization capacity $\alpha_{c}(D_{\textrm{out}}|d_{\textrm{in}})$, we expand both sides of the equations and match the leading divergent terms. The result is:

[TABLE]

with $\alpha_{c}=\alpha_{c}(0|0)$ being the RS capacity. It is $4/\pi$ for the discrete perceptron (which is known to be incorrect) and (correctly) $2$ for the spherical perceptron. Hence, the RSB computation would be needed for the discrete case.

The distance-based capacity $\alpha_{c}(D_{\textrm{out}}|d_{\textrm{in}})$ is manifestly symmetric with respect to

[TABLE]

2.2.3 Analytical and numerical phase diagram

Notably, the limit $Q\to 1$ does not commute with those for $d_{\textrm{in}}\to 0^{+}$ and $d_{\textrm{in}}\to 1^{-}$, where the capacity is discontinuous. Some notable points are

[TABLE]

[TABLE]

[TABLE]

These points show the existence of a critical value $\alpha=\alpha_{c}/3$ below which any degree of generalization is possible. More specifically, from (33) and (35), we see that, below this value, it is typically possible to distinguish arbitrarily similar input sets, unless they are identical. On the other hand, from (34) and (35), we see that it is possible to identify input sets which are arbitrarily different (unless they are exactly anti-parallel).

Moreover, $\alpha_{c}(0|d_{\textrm{in}})$, displays a vertical tangent at $(0,\alpha_{c})$. This means that, when the size of two input sets is close to the typical threshold value $\alpha_{c}N$, then their distance $\Omega$ typically goes to $\infty$ even for small values of their distance $d_{\textrm{in}}$. In this limit, it is typically impossible for a perceptron to identify (give similar outputs) two training sets even if they are very similar.

Fig. 1 shows the agreement between the analytical formula (32) and numerical calculations, in the case of the spherical perceptron.

3 Discussion and Conclusions

The distance-based approach to generalization proposed here is complementary to existing ones, and quantifies how a network can recognize input sets having a prescribed degree of similarity to the training set. This approach corresponds to treating generalization as a task-dependent feature, rather than an absolute property of a neural network. In this framework, a network generalizes well if, given a training set, a typical network configuration assigns similar outputs to input patterns with fixed dissimilarity.

In order to demonstrate the applicability of our approach, we have performed explicit calculations for the perceptron, considering uncorrelated random inputs, with the standard Hamming distance. Our calculations lead to the distance-based generalization capacity $\alpha_{c}(D_{\mathrm{out}}|d_{\mathrm{in}})$. The resulting phase diagram indicates that generalization is possible at any distance, but with decreasing critical capacity. The critical capacity has steeper drops close to the minimal and maximal distances, while showing slower decrease for intermediate increasing distances between the sets of patterns. The formula we have obtained for the critical line, Eq. (32), is formally equivalent to the one computed in a classic study [26] with completely different motivations. The special case $D_{\mathrm{out}}=0$ was very recently rediscovered in a completely different context [27, 28], where it represents the capacity of a perceptron trained to discriminate segments. We surmise that the statistical-mechanics literature on neural networks, spanning the last 40 years, is replete with technical results awaiting to be rediscovered and reinterpreted in more contemporary settings.

The concept of distance-based capacity is general and may be useful in numerical experiments with several kinds of neural networks. Rigorous and computationally feasible definitions of distances for real-world objects are important and challenging, as is well-known, for example, for images [29]. We have chosen here a simple instance of pattern distance but we expect that applying our approach using different system-tailored choices of distance metrics could reveal many aspects of how even complex neural networks operate. The distance-based approach may also be useful for gaining a better understanding of the phenomenon of catastrophic forgetting [30], i.e., the quick increase in the number of errors done on a first training set after the network is trained on a second set.

Finally, we comment briefly about a potential application of $F(D_{\textrm{out}}|d_{\textrm{in}})$ to probe the landscape of synaptic solutions. The existence of difference classes of solutions has been established in the case of the discrete simple perceptron. While dominant solutions are isolated, there exist clustered subdominant solutions, within a certain range [16, 17, 18]. Suppose the numbers $N_{\mathrm{d}}$ and $N_{\mathrm{s}}$ of dominant and subdominant solutions are exponentially large in the size $N$, with different rates $\Sigma_{\mathrm{d}}$ and $\Sigma_{\mathrm{s}}$ such that $\Sigma_{\mathrm{d}}>\Sigma_{\mathrm{s}}$. Then the subdominant class has zero measure in the thermodynamic limit, and is therefore “invisible” to conventional statistical averaging. Now consider the $F(0|d)$ solutions to the problem of learning two input sets $\xi$ and $\bar{\xi}$ at distance $d_{\textrm{in}}$. $\Sigma_{\mathrm{d}}$ and $\Sigma_{\mathrm{s}}$, as functions of $d_{\textrm{in}}$, could cross each other at some value $d_{\textrm{in}}=\tilde{d}_{\textrm{in}}$, so that the subdominant set becomes dominant beyond $\tilde{d}_{\textrm{in}}$. This would be signaled by a discontinuity in $F(0|d_{\textrm{in}})$:

[TABLE]

We speculate that this may be the case if dominant solutions are isolated and more sensitive to small input differences than clustered ones. More in general, non-analytic points of $F(D_{\textrm{out}}|d_{\textrm{in}})$ may reveal a non trivial landscape of solutions, highlighting the presence of different synaptic classes.

Appendix A Replica computation

In this section we will compute $F(D_{\textrm{out}}|d_{\textrm{in}})$ with the replica formalism:

[TABLE]

We can introduce the parameters $Q_{ab}$ and $\hat{Q}_{ab}$ by inserting the identy

[TABLE]

in the integral $\int\prod_{a=1}^{m}dP(W_{a})$. With this choice, $F(D_{\textrm{out}}|d_{\textrm{in}})$ can be rewritten as

[TABLE]

with

[TABLE]

and

[TABLE]

The function $G_{1}$ has two properties:

• •

it depends on $\{W_{a}\}$ only though the overlap $W_{a}\cdot W_{b}/N=Q_{ab}$

• •

it can be rewritten in terms of Gaussian varibles $x_{a}^{\mu}=W_{a}\cdot\xi_{\mu}/\sqrt{N}$.

Both these properties can be deduced from the joint distribution of the auxiliary variables $\{x_{a}^{\mu},\bar{x}_{a}^{\mu}\}_{a=1,...,m}$:

[TABLE]

with

[TABLE]

and $Q$ being a shorthand notation for the matrix $Q_{ab}$. (41) only holds in the thermodynamic limit $N\to\infty$: a key passage consists in expanding $\ln\cosh(y/\sqrt{N})\sim-\frac{y^{2}}{2N}$ for some finte $y$. Therefore

[TABLE]

with

[TABLE]

We can now introduce the RS ansatz

[TABLE]

Under this assumption, we can compute $G_{0}$ and $G_{1}$. The computation of $G_{0}$ is straightforward. In the discrete case,

[TABLE]

If we expand it for small $m$, we get

[TABLE]

In the spherical case,

[TABLE]

with

[TABLE]

For small $m$

[TABLE]

We can compute $G_{1}$ from (43). After some formal manipulations, the functions $A_{\mu}$, as given by (44), can be rewritten as

[TABLE]

with

[TABLE]

and

[TABLE]

For small $m$

[TABLE]

up to $o(m)$. Now, let us observe that

[TABLE]

If we combine (54), (53) and (43), we get (29)

[TABLE]

For the sake of simplicity, we can rewrite (55) as

[TABLE]

Appendix B Saddle point equations and derivation of the critical capacity

From (38) we read the saddle point equations:

[TABLE]

If we use the RS ansatz (45), then (57) and (58) become

[TABLE]

as $m\to 0$. Eq. (60) can be rewritten as (see (A) for the notation)

[TABLE]

Eq. (59) for the discrete model is

[TABLE]

while for the spherical model

[TABLE]

From the previous equations, we can write

[TABLE]

In order to get the critcal capacity, we should evaluate $\alpha(d_{\textrm{in}},D_{\textrm{out}},Q)$ at $Q=1$:

[TABLE]

However, the quantities appearing in the previous equation are neither defined nor analytical at $Q=1$. For this reason we have to extract the leading divergencies around $Q=1^{-}$. We have to expand $D_{\textrm{out}}A_{-}(d_{\textrm{in}},Q)+(1-D_{\textrm{out}})A_{+}(d_{\textrm{in}},Q)$ first. For this purpose, it is convenient to perform a change of variables $S=(y+\bar{y})/\sqrt{2}$, $D=(y-\bar{y})/\sqrt{2}$ (not to be confused with the output difference), $s=(x+\bar{x})/\sqrt{2}$ and write:

[TABLE]

[TABLE]

where we have used the definition $\operatorname{erfc}(y)=\frac{1}{\sqrt{2\pi}}\int_{y}^{\infty}e^{-x^{2}/2}$. We can use the following limit in order to perform the expansion:

[TABLE]

If we set $\epsilon=1-Q$ in (66) and (67), then we obtain that

[TABLE]

where $\mathcal{D}x=dx\;\frac{1}{\sqrt{2\pi}}e^{-x^{2}/2}$, $\chi$ is the characteristic function, $P_{j}$s are second degree polynomials and $\Omega_{j}$s are disjoint sets such that $\cup_{j}\Omega_{j}=\mathbb{R}^{2}$. The explicit computation is a bit lenghty but straightforward and the result is

[TABLE]

However, while $A_{\pm}(Q\to 1)$ (and their derivatives) are finite at both $d_{\textrm{in}}=1$ and $d_{\textrm{in}}=0$, if we fix $Q$, then

[TABLE]

since

[TABLE]

Therefore

[TABLE]

unless $D_{\textrm{out}}=1$, and

[TABLE]

unless $D_{\textrm{out}}=0$. The conclusion is that the limits in $(D_{\textrm{out}},d_{\textrm{in}})$ and $Q$ do not commute. The consequence is the existence of a delta discontinuity. Finally, we can combine (72), (73), (70) and (65) into the distance-based capacity in (32).