Dreaming neural networks: rigorous results

TL;DR

This paper rigorously analyzes a neural network model that mimics sleep and wake cycles, demonstrating how sleep phases enhance memory capacity and stability through advanced mathematical techniques.

Contribution

It extends Guerra's interpolation methods to analyze a sleep-inspired neural network model, confirming its maximal capacity and ergodicity properties within the replica symmetric approximation.

Findings

Sleep phases extend ergodic regions and eliminate spin glass states.

The network achieves maximal storage capacity equal to the number of neurons.

Rigorous mathematical confirmation of the model's properties using Guerra's techniques.

Abstract

Recently a daily routine for associative neural networks has been proposed: the network Hebbian-learns during the awake state (thus behaving as a standard Hopfield model), then, during its sleep state, optimizing information storage, it consolidates pure patterns and removes spurious ones: this forces the synaptic matrix to collapse to the projector one (ultimately approaching the Kanter-Sompolinksy model). This procedure keeps the learning Hebbian-based (a biological must) but, by taking advantage of a (properly stylized) sleep phase, still reaches the maximal critical capacity (for symmetric interactions). So far this emerging picture (as well as the bulk of papers on unlearning techniques) was supported solely by mathematically-challenging routes, e.g. mainly replica-trick analysis and numerical simulations: here we rely extensively on Guerra's interpolation techniques developed for neural networks and, in particular, we extend the generalized stochastic stability approach to the case. Confining our description within the replica symmetric approximation (where the previous ones lie), the picture painted regarding this generalization (and the previously existing variations on theme) is here entirely confirmed. Further, still relying on Guerra's schemes, we develop a systematic fluctuation analysis to check where ergodicity is broken (an analysis entirely absent in previous investigations). We find that, as long as the network is awake, ergodicity is bounded by the Amit-Gutfreund-Sompolinsky critical line (as it should), but, as the network sleeps, sleeping destroys spin glass states by extending both the retrieval as well as the ergodic region: after an entire sleeping session the solely surviving regions are retrieval and ergodic ones and this allows the network to achieve the perfect retrieval regime (the number of storable patterns equals the number of neurons in the network).

Equations

$$H_{N,P}^{\text{(RR)}}(\sigma |\xi ,t):=-\frac{1}{2N}\sum _{i=1}^N\sum _{j=1}^N\sum _{\mu =1}^P\sum _{\nu =1}^P{\xi }_i^{\mu }{\xi }_j^{\nu }{\left(\frac{1+t}{\mathbb{I}+tC}\right)}_{\mu ,\nu }{\sigma }_i{\sigma }_j,$$

$$C_{\mu ,\nu }:=\frac{1}{N}\sum _{i=1}^N{\xi }_i^{\mu }{\xi }_i^{\nu }.$$

$$Z_{N,P}(\sigma |\xi ,t):=\sum _{\{\sigma \}}{e}^{-\beta H_{N,P}^{(RR)}(\sigma |\xi ,t)}=\sum _{\{\sigma \}}\exp \left[\frac{\beta }{2N}\sum _{i,j=1}^{N,N}\sum _{\mu ,\nu =1}^{P,P}{\xi }_i^{\mu }{\xi }_j^{\nu }{\left(\frac{1+t}{\mathbb{I}+tC}\right)}_{\mu ,\nu }{\sigma }_i{\sigma }_j\right].$$

$$⟨O(\sigma ,\xi )⟩$$

$$-\beta F(\alpha ,\beta ,t)\equiv A(\alpha ,\beta ,t):=\underset{N\to \infty }{\lim }\frac{1}{N}\mathbb{E}\ln Z_{N,P}(\sigma |\xi ,t).$$

$$\begin{split}Z_{N,P}(\sigma|\xi,t)&=\sum_{\{\sigma\}}\int\Big{(}\prod_{\mu=1}^{P}d\mu(z_{\mu})\Big{)}\Big{(}\prod_{i=1}^{N}d\mu(\phi_{i})\Big{)}\cdot\\ &\cdot\exp\left(\sqrt{\frac{\beta}{N}(t+1)}\sum_{\mu,i}^{P,N}z_{\mu}\xi^{\mu}_{i}\sigma_{i}+i\sqrt{\frac{t}{N}}\sum_{\mu,i}^{P,N}z_{\mu}\xi^{\mu}_{i}\phi_{i}\right),\end{split}$$

$$H_{N,P}=\frac{a}{\sqrt{N}}\sum _{i=1}^N\sum _{\mu =1}^P{z}_{\mu }{\xi }_i^{\mu }{k}_i,$$

$$a=\sqrt{\beta (t+1)},b=i\sqrt{\frac{t}{\beta (t+1)}}.$$

$$\begin{array}{rl}{q}_{ab}& :=\frac{1}{N}\sum _{i=1}^N{k}_i^{(a)}{k}_i^{(b)},\\ p_{ab}& :=\frac{1}{P}\sum _{\mu \ge 2}{z}_{\mu }^{(a)}{z}_{\mu }^{(b)},\\ m_1& :=\frac{1}{N}\sum _{i=1}^N{\xi }_i^1{k}_i.\end{array}$$

$$\begin{array}{rl}{q}_{ab}& \stackrel{\mathrm{RS}}{\to }W{\delta }_{ab}+q(1-{\delta }_{ab}),\\ p_{ab}& \stackrel{\mathrm{RS}}{\to }X{\delta }_{ab}+p(1-{\delta }_{ab}),\\ m_1& \stackrel{\mathrm{RS}}{\to }m,\end{array}$$

$$\mathcal{A}(s)$$

$$A(\alpha ,\beta ,t)\equiv \underset{N\to \infty }{\lim }\mathcal{A}(s=1)=\underset{N\to \infty }{\lim }\left(\mathcal{A}(s=0)+{\int }_0^1\frac{d\mathcal{A}(s)}{ds}ds\right).$$

$$\begin{split}\frac{d\mathcal{A}(s)}{ds}=&\frac{1}{2N}\operatorname{\mathbb{E}_{\xi,\lambda,\eta}}\Big{[}\frac{a}{\sqrt{sN}}\sum_{i,\mu\geq 2}\xi_{i}^{\mu}\langle z_{\mu}k_{i}\rangle-\frac{1}{\sqrt{1-s}}\Big{(}C_{1}\sum_{i}\eta_{i}\langle k_{i}\rangle+C_{2}\sum_{\mu\geq 2}\lambda_{\mu}\langle z_{\mu}\rangle\Big{)}+\\ &+\frac{a}{\sqrt{sN}}\sum_{i}\xi_{i}^{1}\langle z_{1}k_{i}\rangle-C_{3}\sum_{\mu\geq 2}\langle z_{\mu}^{2}\rangle-C_{4}\sum_{i}\langle k_{i}^{2}\rangle-C_{5}a\sum_{i}\langle\xi_{i}^{1}k_{i}\rangle\Big{]}.\end{split}$$

$$\begin{split}\frac{d\mathcal{A}(s)}{ds}=&\frac{1}{2N}\operatorname{\mathbb{E}_{\xi,\lambda,\eta}}\Big{[}\frac{a^{2}}{N}\sum_{i,\mu\geq 2}\Big{(}\langle z_{\mu}^{2}k_{i}^{2}\rangle-\langle z_{\mu}k_{i}\rangle^{2}\Big{)}+\frac{a^{2}}{N}\langle\big{(}\sum_{i}\xi_{i}^{1}k_{i}\big{)}^{2}\rangle-C_{1}^{2}\sum_{i}\Big{(}\langle k_{i}^{2}\rangle-\langle k_{i}\rangle^{2}\Big{)}\\ &-C_{2}^{2}\sum_{\mu\geq 2}\Big{(}\langle z_{\mu}^{2}\rangle-\langle z_{\mu}\rangle^{2}\Big{)}-C_{3}\sum_{\mu\geq 2}\langle z_{\mu}^{2}\rangle-C_{4}\sum_{i}\langle k_{i}^{2}\rangle-C_{5}a\sum_{i}\langle\xi_{i}^{1}k_{i}\rangle\Big{]}.\end{split}$$

$$\begin{split}\frac{d\mathcal{A}(s)}{ds}=&\frac{1}{2}\operatorname{\mathbb{E}_{\xi,\lambda,\eta}}\Big{[}a^{2}\alpha\langle q_{11}p_{11}\rangle+a^{2}\langle m_{1}^{2}\rangle-a^{2}\alpha\langle q_{12}p_{12}\rangle-C_{1}^{2}\langle q_{11}\rangle+C_{1}^{2}\langle q_{12}\rangle+\\ &-C_{2}^{2}\alpha\langle p_{11}\rangle+C_{2}^{2}\alpha\langle p_{12}\rangle-\alpha C_{3}\langle p_{11}\rangle-C_{4}\langle q_{11}\rangle-aC_{5}\langle m_{1}\rangle\Big{]}.\end{split}$$

$$C_1^2=a^2\alpha p,C_2^2=a^{2}q,C_3=a^2(W-q),C_4=a^2\alpha (X-p),C_5=2ma,$$

$$\begin{split}\frac{d\mathcal{A}(s)}{ds}=&\frac{1}{2}\operatorname{\mathbb{E}_{\xi,\lambda,\eta}}\Big{[}a^{2}\alpha\langle(q_{11}-W)(p_{11}-X)\rangle+a^{2}\langle(m_{1}-m)^{2}\rangle-a^{2}\alpha\langle(q_{12}-q)(p_{12}-p)\rangle\Big{]}+\\ &+\frac{\alpha a^{2}}{2}(qp-WX)-\frac{a^{2}}{2}m^{2}.\end{split}$$

$$\begin{array}{r}{d}_s\mathcal{A}(s)=\frac{\alpha a^2}{2}(qp-WX)-\frac{a^2}{2}{m}^2\end{array}$$

$$\begin{split}\mathcal{A}(s=0)=&\frac{1}{N}\operatorname{\mathbb{E}_{\xi,\eta,\lambda}}\ln\sum_{\sigma}\int\!\operatorname{d\mu}\left(z,\phi\right)\exp\Big{[}C_{1}\sum_{i}\eta_{i}k_{i}+\frac{C_{4}}{2}\sum_{i}k_{i}^{2}+\frac{C_{5}a}{2}\sum_{i}\xi_{i}^{1}k_{i}+\\ &+C_{2}\sum_{\mu\geq 2}\lambda_{\mu}z_{\mu}+\frac{C_{3}}{2}\sum_{\mu\geq 2}z_{\mu}^{2}\Big{]}.\end{split}$$

$$\begin{split}\mathcal{A}(s=0)=&-\frac{\alpha}{2}\ln(1-C_{3})-\frac{1}{2}\ln(1-C_{4}b^{2})+\frac{\alpha}{2}\frac{C_{2}^{2}}{1-C_{3}}+\frac{C_{4}}{2}+\operatorname{\mathbb{E}_{\eta}}\ln\cosh\Big{[}\frac{C_{1}\eta+\frac{C_{5}a}{2}}{1-C_{4}b^{2}}\Big{]}+\\ &+b^{2}\frac{C_{1}^{2}+C_{4}^{2}+\frac{C_{5}^{2}a^{2}}{4}}{1-C_{4}b^{2}}+\ln 2.\end{split}$$

$$\begin{split}A_{{\rm\scriptscriptstyle RS}}(\alpha,\beta,t)=&\frac{\alpha a^{2}}{2}\big{(}qp-WX\big{)}-\frac{a^{2}}{2}m^{2}-\frac{\alpha}{2}\ln\big{[}1-a^{2}(W-q)\big{]}-\frac{1}{2}\ln\big{[}1-a^{2}b^{2}\alpha(X-p)\big{]}+\\ &+\frac{\alpha}{2}\frac{a^{2}q}{1-a^{2}(W-q)}+\frac{\alpha a^{2}}{2}\big{(}X-p\big{)}+\frac{a^{2}b^{2}}{2}\cdot\frac{\alpha p+m^{2}a^{2}+a^{2}\alpha^{2}(X-p)^{2}}{1-a^{2}b^{2}\alpha(X-p)}+\\ &+\ln 2+\operatorname{\mathbb{E}_{\eta}}\ln\cosh\Big{[}\frac{a\eta\sqrt{\alpha p}+ma^{2}}{1-\alpha a^{2}b^{2}(X-p)}\Big{]}.\end{split}$$

$$X\to \frac{{\beta }^2}{a^2}X,p\to \frac{{\beta }^2}{a^2}p,m\to \frac{\beta }{a^2}m,$$

$$\begin{split}F_{{\rm\scriptscriptstyle RS}}(\alpha,\beta,t)=&-\frac{\beta m^{2}}{2(1+t)}\Big{(}1+\frac{t}{\Delta}\Big{)}-\frac{(1+t)(\Delta-1)}{2t}\beta W-\frac{\alpha\beta^{2}}{2}p(W-q)\\ &-\frac{\alpha}{2}\Big{(}\log[1-\beta(1+t)(W-q)]+\frac{q\beta^{2}(1+t)}{1-\beta(1+t)(W-q)}\Big{)}-\frac{(1+t)(1-\Delta)\beta}{2t\Delta}\\ &-\frac{\log\Delta}{2}-\frac{\alpha\beta pt}{2(1+t)\Delta}+\operatorname{\mathbb{E}_{\eta}}\log\cosh\Big{[}\frac{\beta}{\Delta}(m+\sqrt{\alpha p}\eta)\Big{]}+\log 2.\end{split}$$

$$\begin{split}m&=\frac{1+t}{\Delta+t}\operatorname{\mathbb{E}_{\eta}}\tanh\Big{[}\frac{\beta}{\Delta}(m+\sqrt{\alpha p}\eta)\Big{]},\\ p&=\frac{q(1+t)^{2}}{[1-\beta(1+t)(W-q)]^{2}},\\ \Delta&=1+\frac{\alpha t}{1-\beta(1+t)(W-q)},\\ q&=W+\frac{t}{\beta(1+t)\Delta}-\frac{1}{\Delta^{2}}\operatorname{\mathbb{E}_{\eta}}\cosh^{-2}\Big{[}\frac{\beta}{\Delta}(m+\sqrt{\alpha p}\eta)\Big{]},\\ W\Delta^{2}&=1-\frac{t\Delta}{\beta(1+t)}+\frac{\alpha pt^{2}-m^{2}t(t+2\Delta)}{(1+t)^{2}}-\frac{2\alpha\beta pt}{(1+t)\Delta}\operatorname{\mathbb{E}_{\eta}}\cosh^{-2}\Big{[}\frac{\beta}{\Delta}(m+\sqrt{\alpha p}\eta)\Big{]}.\end{split}$$

$$\begin{split}\theta_{lm}&=\sqrt{N}\big{[}q_{lm}-\delta_{lm}W-(1-\delta_{lm})q\big{]}\\ \rho_{lm}&=\sqrt{P}\big{[}p_{lm}-\delta_{lm}X-(1-\delta_{lm})p\big{]}.\end{split}$$

$$\begin{split}\mathcal{A}^{r}_{J}(s)=&\operatorname{\mathbb{E}_{\xi,\eta,\lambda}}\ln\sum_{\sigma_{R}}\int\!\operatorname{d\mu}\left(z_{R},\phi_{R}\right)\exp\Big{[}\sqrt{s}\frac{a}{\sqrt{N}}\sum_{l=1}^{r}\sum_{i,\mu}z_{\mu}^{(l)}\xi_{i}^{\mu}k_{i}^{(l)}+J\hat{O}\\ &+\sqrt{1-s}\Big{(}C_{1}\sum_{l=1}^{r}\sum_{i}\eta_{i}k_{i}^{(l)}+C_{2}\sum_{l=1}^{r}\sum_{\mu}\lambda_{\mu}z_{\mu}^{(l)}\Big{)}+\frac{1-s}{2}\Big{(}C_{3}\sum_{l=1}^{r}\sum_{\mu}(z_{\mu}^{(l)})^{2}+C_{4}\sum_{l=1}^{r}\sum_{i}(k_{i}^{(l)})^{2}\Big{)}\Big{]}.\end{split}$$

$$⟨O(s)⟩={\frac{\partial {\mathcal{A}}_J^r(s)}{\partial J}|}_{J=0},{\partial }_s⟨O(s)⟩={\frac{\partial ({\partial }_s{\mathcal{A}}_J^r)}{\partial J}|}_{J=0}.$$

$$\partial_{s}\mathcal{A}^{r}_{J}=\frac{1}{2}\sqrt{\alpha}\beta(1+t)\sum_{l,m=1}^{r}\Big{[}\langle g_{l,m}\rangle-\langle g_{l,m+r}\rangle\Big{]},\quad\quad g_{l,m}=\theta_{l,m}\rho_{l,m}.$$

$$(X,p)\to \frac{{\beta }^2}{a^2}(X,p).$$

$$d_{\tau }⟨O⟩=\frac{1}{2}\sum _{a,b}^r⟨O\cdot g_{a,b}⟩-r\sum _{a=1}^r⟨O\cdot g_{a,r+1}⟩+\frac{r(r+1)}{2}⟨O\cdot g_{r+1,r+2}⟩-\frac{r}{2}⟨O\cdot g_{r+1,r+1}⟩,$$

Full text

So far this emerging picture (as well as the bulk of papers on unlearning techniques) was supported solely by mathematically-challenging routes, e.g. mainly replica-trick analysis and numerical simulations: here we rely extensively on Guerra’s interpolation techniques developed for neural networks and, in particular, we extend the generalized stochastic stability approach to the case. Confining our description within the replica symmetric approximation (where the previous ones lie), the picture painted regarding this generalization (and the previously existing variations on theme) is here entirely confirmed.

Further, still relying on Guerra’s schemes, we develop a systematic fluctuation analysis to check where ergodicity is broken (an analysis entirely absent in previous investigations). Remarkably we find that, as long as the network is awake, ergodicity is bounded by the Amit-Gutfreund-Sompolinsky critical line (as it should), but, as the network sleeps, sleeping destroys spin glass states by extending both the retrieval as well as the ergodic region: after an entire sleeping session the solely surviving regions are retrieval and ergodic ones and this allows the network to achieve the perfect retrieval regime (where the number of storable patterns exactly equals the number of neurons the network is built of).

1 Introduction

Statistical mechanics of spin glasses MPV has been playing a primary role in the investigation of neural networks, as for the description of both their learning phase angel-learning ; sompo-learning and their retrieval properties Amit ; Coolen . Along the past decades, beyond the bulk of results achieved via the so-called replica-trick MPV , a considerable amount of rigorous results exploiting alternative routes (possibly mathematically more transparent) were also developed (see e.g. Agliari-Barattolo ; ABT ; Bovier1 ; Bovier2 ; Bovier3 ; Albert1 ; Barra-JSP2010 ; bipartiti ; Dotsenko1 ; Dotsenko2 ; Tala1 ; Tala2 ; Tirozzi ; Pastur and references therein). This paper goes in the latter direction and focuses on a generalization of the Hopfield model Albert2 that is able to saturate the optimal storage capacity and whose main characteristics are summarized hereafter.

In Albert2 the Hebbian kernel underlying the Hopfield model was revised to account also for reinforcement and removal processes. The resulting kernel can be interpreted as the effect of a daily routine: during the awake state, the network is fed with inputs (i.e. patterns of information) that are stored in an Hebbian fashion111We stress that, given the equivalence between restricted Boltzmann machines and Hopfield neural networks BarraEquivalenceRBMeAHN , also learning via e.g. contrastive divergence Hinton1 ultimately falls into the Hebbian category Agliari-Dantoni ; Agliari-Isopi ., then, during the asleep state, it weeds out the (combinatorial222The growth in the number of spurious states is roughly exponential in the number of stored patterns, namely -in the high storage regime- in the number of neurons.) proliferation of the spurious mixtures (unavoidably created as metastable states in the free-energy landscape of the system during the learning stage) and it consolidates the pure states (making their free-energy minima deeper in this landscape picture). Remarkably, after these procedures, the network is able to saturate the storage capacity $\alpha$ (that is the amount of stored patterns $P$ over the amount of available neurons $N$, in the thermodynamic limit, i.e. $\alpha=\lim_{N\to\infty}P/N$) to its upper bound333Actually the network seems to perform even better, returning its maximal capacity to be $\alpha_{c}\sim 1.07>1$: this is obviously not possible and, as explained by Dotsenko and Tirozzi Dotsenko1 ; Dotsenko2 , it is a chimera of the replica-symmetric regime at which the theory is developed. which, for symmetric networks, is $\alpha_{c}=1$ Gardner . Further, in the retrieval phase of its parameter space, pure states are global minima up to $\alpha\sim 0.85$ (see Figure 1), that is a much broader range with respect to the classical Hopfield counterpart, where they remain global minima solely for $\alpha<0.05$.

In this work, we first show the equivalence between the aforementioned generalized neural network and a tripartite (or “three-layers” in a machine-learning jargon) spin-glass, where couplings between neurons of different layers exhibit correlations and the third layer is a spectral layer equipped with imaginary numbers (see Fig. 2 and Remark 3). Then, we generalize the stochastic stability technique, introduced in AC1 ; CG1 to address Sherrington-Kirkpatrick spin-glass and later developed in Barra-JSP2010 to account also for bipartite spin-glasses (namely restricted Boltzmann machines or Hopfield networks BarraEquivalenceRBMeAHN in a machine learning jargon DLbook ; Hugo ), so that it can as well deal with the present tripartite and correlated spin-glass.

Next, by using this novel approach -that is mathematically well controllable at any stage of the calculations- we obtain the expression of the quenched replica-symmetric free energy related to the model (as well as the set of self-consistent equations for the order parameters) and we show that the resulting picture sharply coincides with that obtained via the replica-trick analysis Albert2 . This implies, in a cascade fashion, that all the results previously heuristically derived are actually proved (the most remarkable one being the saturation of the critical capacity).

Finally, we extend our analysis to order-parameter fluctuations in order to investigate ergodicity breaking: interestingly, as suggested also by the self-consistencies, we find that -without sleeping- ergodicity breaks as predicted by Amit-Gutfreund-Sompolinsky Amit (as it should), but -as sleeping takes place- the spin-glass region shrinks and ultimately the network phase-diagram exhibits only retrieval and ergodic phases (see Fig.s 5,6).

This paper is structured as follows: in Sec. $2$, once the model is introduced and embedded in its statistical mechanical framework, we calculate its quenched free energy by introducing a novel interpolating structure à la Guerra and this provides a first picture of the phase diagram of the model (as we can identify the transition between the retrieval and the spin-glass regions). Next, in Sec. $3$, we study the fluctuations of the order parameters to inspect where ergodicity is spontaneously broken as this is a signature of the critical line, namely the transition between the ergodic and the spin-glass regions): by combining the two results a full picture of the phase diagram of the model can be finally deduced. Sec. $4$ is left for conclusions. Technical details and further remarks on the interpolation approach are provided in the appendices.

2 Replica symmetric free energy analysis

2.1 Definition of the Model

Driven by the works of Personnaz, Guyon, Dreyfus Personnaz and of Dotsenko et al. Dotsenko1 ; Dotsenko2 , in Albert2 we introduced the following generalization of the standard Hopfield paradigma Hopfield , referred to as “reinforcement&removal” (RR) algorithm: consider a network composed by $N$ Ising neurons $\{\sigma_{i}\}_{i=1,...,N}$ and $P$ patterns $\{\xi^{\mu}\}_{\mu=1,...,P}$ (namely quenched random vectors of the same length $N$), and denote with $t\in\mathbb{R}^{+}$ the sleep extent (such that for $t=0$ the network has never slept, while for $t\to\infty$ an entire sleeping session has occurred), we can then introduce the following

Definition 1**.**

The Hamiltonian of the reinforcement$\&$removal model reads as:444As a matter of notation, we stress that the denominator $1/(\mathbb{I}+tC)$ in the generalized kernel is intended as the inverse matrix $(\mathbb{I}+tC)^{-1}$.

[TABLE]

where $\sigma_{i}=\pm 1\ \forall i\in(1,...,N)$, $\xi^{1}$ -that is the pattern candidate to be retrieved- has binary entries $\xi_{i}^{1}\in\{-1,+1\}$ drawn from $P(\xi_{i}^{\mu}=+1)=P(\xi_{i}^{\mu}=-1)=\frac{1}{2}$, while the remaining $P-1$ patterns $\{\xi^{\mu}\}_{\mu=2,...,P}$, have i.i.d. standard Gaussian entries $\xi_{i}^{\mu}\sim\mathcal{N}[0,1]$, and the correlation matrix $\boldsymbol{C}$ is defined as

[TABLE]

Remark 1**.**

*We stress that, for the sake of mathematical convenience, as deepened in Agliari-Barattolo , we take solely the pattern candidate for retrieval (i.e. the signal) to be Boolean, while all the remaining ones (acting as slow noise on the retrieval) are chosen as Gaussian: although neural networks, in general, do not exhibit the universality properties of spin glasses Genovese , this is no longer true if we confine our focus solely to the structure of the slow noise generated by patterns555As extensively discussed in Barra-RBMsPriors1 ; Barra-RBMsPriors2 by varying the nature of the neurons as well as of the pattern entries, for instance ranging from Boolean (Ising) to standard Gaussians, the retrieval performances of the network vary sensibly and, in some limits, are entirely lost: in this sense neural networks do not share universality with standard spin-glasses.. *

Remark 2**.**

*Note that the matrix ${\xi}^{T}\left(\frac{1+t}{\mathbb{I}+t{C}}\right){\xi}$, encoding the neuronal coupling, recovers the Hebbian kernel for $t=0$ , while it approaches the pseudo-inverse matrix for $t\rightarrow\infty$ (see Albert2 for the proof). Accordingly, the model described by the Hamiltonian (1) spans, respectively, from the standard Hopfield model $(t\to 0)$ to the Kanter-Sompolinksy model KanterSompo $(t\to\infty)$.

During the sleeping session, both reinforcement and remotion take place: oversimplifying, in the generalized synaptic coupling appearing in (1), the denominator (i.e., the term $\propto(1+tC)^{-1}$) yields to the remotion of unwanted mixture states, while the numerator (i.e., the term $\propto 1+t$) reinforces the pure memories.*

We are interested in obtaining the phase diagram of the model coded by the cost function (1), solely in the thermodynamic limit and under the replica symmetric assumption. To achieve this goal the following definitions are in order.

Definition 2**.**

Using $\beta\in\mathbb{R}^{+}$ as a parameter tuning the level of fast noise in the network (with the physical meaning of inverse temperature, i.e. calling $T$ the temperature, $\beta\equiv T^{-1}$ in proper units,), the partition function of the model (1) is introduced as

[TABLE]

Definition 3**.**

Denoting with $\mathbb{E}_{\xi}$ the average over the quenched patterns, for a generic function $O(\sigma,\xi)$ of the neurons and the couplings, we can define the Boltzmann $\langle O(\sigma,\xi)\rangle$ as

[TABLE]

such that its quenched average reads as $\mathbb{E}_{\xi}\langle O(\sigma,\xi)\rangle$.

Definition 4**.**

Once introduced the partition function $Z_{N,P}(\sigma|\xi,t)$, we can define the infinite volume limit of the intensive quenched free-energy $F_{N}(\alpha,\beta,t)$ and of the intensive quenched pressure $A(\alpha,\beta,t)$ associated to the model (1) as

[TABLE]

As anticipated, the pressure of the model (1) was analyzed in Albert2 via replica-trick Coolen (corroborated by extensive numerical simulations), showing that (at the replica symmetric level of description) the maximal critical capacity of this neural network saturates the Gardner’s bound Gardner (i.e. $\alpha_{c}=1$, for symmetric noiseless networks).

Remark 3**.**

The partition function defined in (2) can be represented in Gaussian integral form as

[TABLE]

where $d\mu(z_{\mu})$ and $d\mu(\phi_{i})$ are the standard Gaussian measures. This relation shows that the partition function of the reinforcement$\&$removal model is equivalent to the partition function of a tripartite spin-glass where the intermediate party (or hidden layer to keep a machine learning jargon) is made of real neurons $\{z_{\mu}\}_{\mu=1,...,P}$ with $z_{\mu}\sim\mathcal{N}[0,1],\forall\mu$, while the external layers are made, respectively, of a set of Boolean neurons $\{\sigma_{i}\}_{i=1,...,N}$ (the visible layer) and of a set of imaginary neurons with magnitude $\{\phi\}_{i=1,...,N}$, being $\phi_{i}\sim\mathcal{N}[0,1],\forall i$ (the spectral layer), see Fig. 2.

2.2 Guerra’s interpolating framework for the free energy

Definition 5**.**

Once expressed the partition function (2) in its integral representation (6), we can introduce the related tripartite spin glass Hamiltonian as

[TABLE]

where we introduced the “multi-spin” $k_{i}=\sigma_{i}+b\phi_{i}$ and where

[TABLE]

Remark 4**.**

Note that the cost function (7) and the one associated to the original model (1) share the same partition function and therefore exhibit the same Thermodynamics. By a practical perspective, the latter is more suitable for understanding the retrieval capabilities of the network, the former for dealing with its learning skills BarraEquivalenceRBMeAHN ; Barra-RBMsPriors1 .

In the following we consider the challenging case with $P=\alpha N$ for large $N$ and we aim to obtain an expression for the quenched pressure (5) in terms of the order parameters introduced in the next

Definition 6**.**

The natural order parameters for the neural network model (1) -as suggested by its integral representation (7)- are the overlaps $q_{ab}$ and $p_{ab}$ between the $k$’s and the $z$’s variables, respectively, as functions of two replicas (a,b) of the system, and the generalized Mattis overlap666We arbitrarily (but with no loss of generality) nominated the first pattern as the retrieved one. $m_{1}$, namely

[TABLE]

Remark 5**.**

The replica symmetric approximation (RS) is imposed by requiring that the order-parameters of the theory do not fluctuate in the thermodynamic limit777This request is obviously perfectly consistent with the replica-symmetric ansatz when approaching the problem via the replica trick Coolen ; Albert2 ., i.e.

[TABLE]

where we called, respectively, $W,q,X,p,m$ the replica symmetric values of the diagonal and off-diagonal overlap $q$, the diagonal and off-diagonal overlap $p$ and the Mattis magnetization $m_{1}$.

Now the plan is to get an explicit expression for the pressure (5) in terms of these order parameters, to extremize the former over the latter and get a phase diagram for the network. To reach this goal we generalize a Guerra’s interpolation scheme Barra-JSP2010 : the idea is to compare the original system, as represented in eq. (7) (namely a three-layer correlated spin glass), with three random single-layers, where each layer experiences, statistically, the same mean-field that would have been produced by the other layers over it. To this aim we introduce the following

Definition 7**.**

Being $s\in[0,1]$ an interpolating parameter, $\{\eta_{i}\}_{i\in(1,...,N)}$ a set of $N$ i.i.d. Gaussian variables, $\{\lambda_{\mu}\}_{\mu\in(2,...,P)}$ a set of $P-1$ i.i.d. Gaussian variables, and the scalars $C_{1},C_{2},C_{3},C_{4},C_{5}$ to be set a posteriori, we use as interpolating pressure the following quantity

[TABLE]

Remark 6**.**

When $s=1$ we recover the original model, namely $A(\alpha,\beta,t)=\lim_{N\to\infty}\mathcal{A}(s=1)$, while for $s\to 0$ we are left with a one-body problem, and, consequently, the probabilistic structure of $\mathcal{A}(s=0)$ is more tractable.

Remark 7**.**

We note the importance of splitting the sum on the $\xi$’s into $\xi^{1}$ (i.e. the signal) and the $\xi^{2}\cdots\xi^{P}$ (i.e. the quenched noise) since the quenched average treats them differently, and so we will need to address them separately.

Proposition 1**.**

The infinite volume limit of the quenched pressure related to the model (1) can be obtained by using the Fundamental Theorem of Calculus as

[TABLE]

To follow this approach, two calculations are in order: the streaming $d_{s}\mathcal{A}(s)$ (and its successive back-integration) and the evaluation of the Cauchy condition $\mathcal{A}(s=0)$. Let us start with $d_{s}\mathcal{A}(s)$:

[TABLE]

We can proceed further by using Wick’s Theorem [$\mathbb{E}_{x}xF(x)=\mathbb{E}_{x}(x^{2})\cdot\mathbb{E}_{x}\partial_{x}F(x)$] on the fields $z^{1},\,\,\xi^{2\cdots P},\,\,\lambda_{\mu},\,\,\eta_{i}$, obtaining

[TABLE]

Using the definition of the order parameters (9) we can write $d_{s}\mathcal{A}(s)$ as

[TABLE]

It is now convenient to fix the free scalars $C_{1,..,5}$ as

[TABLE]

such that we can recast the streaming $d_{s}\mathcal{A}(s)$ as

[TABLE]

Remark 8**.**

When requiring replica symmetry, we have that $\langle q_{11}\rangle\to W$, $\langle p_{11}\rangle\to X$, $\langle m_{1}\rangle\to m$, $\langle q_{12}\rangle\to q$ and $\langle p_{12}\rangle\to p$, hence the evaluation of the integral in eq. (12) becomes trivial as the r.h.s. of eq. (17) reduces to

[TABLE]

that does not depend on $s$ any longer.

We must now evaluate the one-body contribution $\mathcal{A}(s=0)$: this can be done by directly setting $s=0$ in (7)

[TABLE]

Performing standard Gaussian integrations we obtain

[TABLE]

Keeping in mind the expressions for the parameters $C_{1},...,C_{5}$ as prescribed in the relations 16, by plugging eq. (18) and eq. (20) into the sum rule (12) we finally get an expression for the quenched pressure of the model (1) in terms of the replica-symmetric order parameters

[TABLE]

To match exactly the notation in Albert2 there is still a short way to go: it is convenient to re-scale $m$, $p$ and $X$ as

[TABLE]

as this allows us to introduce the composite order parameter $\Delta=1-\alpha\beta^{2}b^{2}(X-p)$ used in Albert2 .

After these transformations, remembering the definition of the free energy (see (5)) and the definition of $(a,b)$ (see (8)), we obtain exactly the same expression for the quenched free energy as that achieved in Albert2 via the replica trick, as stated by the next main

Theorem 1**.**

In the infinite volume limit, the replica symmetric free energy related to the neural network defined by eq. (1) can be expressed in terms of the natural order parameters of the theory (see def.s (9)) as

[TABLE]

Proposition 2**.**

Using the standard variational principle $\vec{\nabla}F_{{\rm\scriptscriptstyle RS}}=0$ on the free energy (23), namely by extremizing the latter over the order parameters, we obtain the following set of self-consistent equations for these parameters, whose behavior is outlined in the plots of Fig. 3.

[TABLE]

Remark 9**.**

We stress that we obtained exactly the same self-consistencies previously appeared in Albert2 , thus all the consequences stemming by them, as reported in that paper, are here entirely confirmed.

3 Study of the overlap fluctuations

As proved in the previous section, the reinforcement$\&$removal algorithm makes the retrieval region in the $(\alpha,\beta)$ plane wider and wider as $t$ is increased (see Fig. 1). As the retrieval region pervades the spin-glass region, one therefore naturally wonders whether the opposite boundary of the spin-glass region (namely the critical line depicting the transition where ergodicity breakdowns) is as well deformed. To address this point, we now study the behavior of the overlap fluctuations, suitably centered around the thermodynamic values of the overlaps and properly rescaled in order to allow them to diverge when the system approaches the critical line. In fact, they are meromorphic functions and their poles identify the evolution of the critical surface $\beta_{c}(\alpha,t)$ (if any).

It is worth recalling that the critical line for the standard Hopfield model Hopfield as predicted by the AGS theory Amit is $\beta_{c}(\alpha,t=0)=(1+\sqrt{\alpha})^{-1}$.

3.1 Guerra’s interpolating framework for the overlap fluctuations

The idea is the same exploited in the previous section, namely to use the generalized Guerra’s interpolation scheme (see eq. (7)) to evaluate the evolution of the order parameter’s correlation functions from $s=0$ (where they do not represent the real fluctuations in the system, but their evaluation should be possible) up to $s=1$ (where they reproduce the true fluctuations). To achieve this goal for the generic correlation function $O$, we need to evaluate the Cauchy condition $\langle O(s=0)\rangle$ and the derivative $\partial_{s}\langle O(s)\rangle$. However, in contrast with the previous section where we imposed replica symmetry, here -as we just want to infer the critical line- we impose ergodic behavior, namely, we assume that the system is approaching this boundary from the high fast-noise limit. This allows us to set all the mean values of the overlaps to zero and to achieve explicit solutions.

Definition 8**.**

The centered and rescaled overlap fluctuations $\theta_{lm}$ and $\rho_{lm}$ are introduced as

[TABLE]

Remark 10**.**

As we will address the problem of the overlap fluctuations in the ergodic region, the signal is absent, thus there is no need to introduce a rescaled Mattis order parameter: only the boundary between the ergodic region and the spin-glass region is under study here.

Proposition 3**.**

It is convenient to introduce the $r-$replicated interpolating pressure $\mathcal{A}_{J}^{r}(s)$, where we further added a source field $J$, coupled to an observable $O$ (that is a smooth function of the neurons of the $r$-replicas) as

[TABLE]

where $k_{i}$ is the same as in Definition $5$ and the interpolation constants $C_{1,2,3,4}$ are the same given in the previous section (see eq. ((16))).

By definition

[TABLE]

Therefore, in order to evaluate the fluctuations of $O$ we need to evaluate first $\partial_{s}\mathcal{A}^{r}_{J}$ and, by a routine calculation, we get

[TABLE]

To evaluate the fluctuations of a general operator $O$, function of $r-$replicas, we must use the results (27) and perform the same rescaling that we did in the previous section, namely

[TABLE]

Overall this brings to the next

Proposition 4**.**

Given $O$ as a smooth function of $r$ replica overlaps $\left(q_{1},\ldots,q_{r}\right)$ and $\left(p_{1},\ldots,p_{r}\right),$ the following streaming equation holds:

[TABLE]

where we used the operator $d_{\tau}$ defined as

[TABLE]

in order to simplify calculations and presentation.

3.2 Criticality and ergodicity breaking

To study the overlap fluctuations we must consider the following correlation functions (it is useful to introduce and link them to capital letters in order to simplify their visualization):

[TABLE]

Since we are interested in finding the critical line for ergodicity breaking from above we can treat $\theta_{a,b},\rho_{a,b}$ as Gaussian variables with zero mean (this allows us to apply Wick-Isserlis theorem inside averages) as we can also treat both the $k_{i}$ and $z_{\mu}$ as zero mean random variables in the ergodic region (thus all averages involving uncoupled fields are vanishing): this considerably simplifies the evaluation of the critical line (as expected since we are approaching criticality from the trivial ergodic region BarraGuerra-JMP2008 ).

We can thus reduce the analysis to

[TABLE]

According to (30) and to the previous reasoning we obtain:

[TABLE]

Suitably combining $A$ and $G$ in (34) we can write

[TABLE]

Now we are left with

[TABLE]

The trick here is to complete the square by summing $d_{\tau}D+rd_{\tau}G$ thus obtaining

[TABLE]

The solution is trivial and it is given by

[TABLE]

So we are left with the evaluation of the correlations at $s=0$: namely the Cauchy conditions related to the solution coded in eq. (38). To this task we introduce a one-body generating function for the momenta of $z,k$: this can be done by setting inside (26) $s=0,r=1$ and adding source fields $(j_{i},J_{\mu})$ coupled respectively to $(k_{i},z_{\mu})$, with $i\in(1,...,N),\,\mu\in(1,...,P)$. Since we are approaching the critical line from the high fast noise limit we can set $m,p,q=0$ (when we explicitly make use of the coefficients (16)), overall writing

[TABLE]

Clearly, we took great advantage in approaching the ergodic region from above, since even the one-body problem (for the Cauchy condition) has been drastically simplified: showing only the relevant terms in $j,J$ we have

[TABLE]

As anticipated, all the observable averages needed at $s=0$ can now be calculated simply as derivatives of $F(j,J)$, thus the $s=0$ correlation functions are finally given by

[TABLE]

Inserting this result in (38), we get

[TABLE]

Upon evaluating $Y(\tau)$ for $\tau=\beta(1+t)\sqrt{\alpha}s,\,s=1$ and reporting the relevant ergodic self-consistent equations we obtain the following system:

[TABLE]

Since we are interested in obtaining the critical temperature for ergodicity breaking, where fluctuations (in this case $Y$) grow arbitrarily large we can check where the denominator at the r.h.s. of the first eq. (43) becomes zero and recast this observation as follows

Theorem 2**.**

The ergodic region of the model defined by the cost function (1) is delimited by the following critical surface in the $(\alpha,\beta,t)$ space of the tunable parameters

[TABLE]

Remark 11**.**

*At $t=0$, where the model reduces to Hopfield’s scenario, the critical surface correctly collapses over the Amit-Gutfreund-Sompolinsky critical line $\beta_{c}=(1+\sqrt{\alpha})^{-1}$, but in the large $t$ limit the ergodic region collapses to the axis $T=0$: this may have a profound implication, namely that the ergodic region -during the sleep state- phagocytes the spin-glass region.

Since we have already seen that also the retrieval region phagocytes the spin-glass region 888Note that the ergodic line does not affect the retrieval region, they simply fade one into the other. This is because the critical surface is calculated assuming an ergodic regime (hence, it does not takes into account the signal) and, more importantly, the retrieval region is delimited by a first order phase transition, that is not detected by a second order inspection as that needed for criticality. this means that spurious states are entirely suppressed with a proper rest, allowing the network to achieve perfect retrieval, as suggested in the pioneering study by Kanter and Sompolinsky KanterSompo .*

4 Conclusions and outlooks

In recent years Artificial Intelligence, mainly due to the impressive skills of Deep Learning machines and the GPU-related revolution DL1 , has attracted the attention of the whole Scientific Community. In particular, the latter includes mathematicians involved in the statistical mechanics of complex systems which has proved to be a fruitful tool in the investigation of neural networks and machine learning, since the early days (not by chance Boltzmann machines are named after Boltzmann BM1 ).

Among the various fields of Artificial Intelligence where, in the present years, statistical mechanics extensively contributed to the cause (e.g. statistical inference and signal processing Simona ; Lenka1 , combinatorial and computational complexity Lenka2 ; Zecchina ; Monasson , supervised or unsupervised learning Zecchina2 ; Huang , deep learning Chiara ; Metha , compositional capabilities Agliari-PRL1 ; Monasson2 , and really much more…) the one we deepened in this work deals with the phenomenon of dreaming and sleeping999We point out that dreaming has been recently connected to compositional capabilities Guardian , the latter being natural properties of diluted retricted Boltzmann machines Agliari-Dantoni ; Agliari-Immune ; Monasson2 ..

In the current work we mathematically described the phenomena of reinforcement and remotion, as pioneered by Crick $\&$ Mitchinson Crick , by Hopfield HopfieldUnlearning and by many others in the neuroscience literature, see e.g Neuro-1 ; Neuro0 ; Neuro1 ; Neuro2 ): interestingly, such mechanisms have been evidenced to lead to an improvement of the retrieval capacity of the system. In particular, in Albert2 , we showed that the system reaches the expected upper critical capacity $\alpha_{c}=1$, still preserving robustness with respect to fast noise. However, the statistical mechanical analysis, set at the standard replica symmetric level of description, was carried out via non-rigorous approaches (e.g., replica trick and numerical simulations).

In this work we extended a Guerra’s interpolation scheme Barra-JSP2010 , originally developed to deal with the standard Hopfield model (i.e. equipped with the canonical Hebbian synaptic coupling), to deal with this generalization: at first we showed the equivalence of this model with a three-layer spin-glass where some links among different layers are cloned (hence introducing correlation in the network and in the random fields required for the interpolation) and the third, and novel (w.r.t. the standard equivalence between Hopfield models and two-layers Boltzmann machines BarraEquivalenceRBMeAHN ; Barra-RBMsPriors2 ), layer is equipped with imaginary real-valued neurons (best suitable to perform spectral analysis101010We plan to report soon on the learning algorithms for this generalized restricted Boltzmann machine, where the properties of the spectral layers will spontaneously shine.). As a consequence, the resulting interpolating architecture is rather tricky, by far richer than its classical limit yet it turns out to be managable and actually a sum rule for the quenched free energy related to the model can be written and even integrated, under the assumption of replica symmetry: such an expression, as well as those stemming from its extremization for the order parameters, sharply coincides with previous results Albert2 , confirming them in each detail.

We remark that such theorems state also the validity of other previous investigation -all replica trick derived- on unlearning in neural networks (see e.g. Dotsenko1 ; unlearning1 ; KanterSompo ).

Beyond confirming previous results, we further systematically developed a fluctuation analysis of the overlap correlation functions, searching for critical behaviour, in order to inspect where ergodicity breaks down and in this investigation we found a very interesting result: as long as the Hopfield model is awake, the critical line is the one predicted by Amit-Gutfreund-Sompolinksy (as it should and as it is known by decades). However, as the network sleeps, the ergodic region starts to invade the spin glass region, ultimately destroying the spin glass states entirely, thus allowing the network (at the end of an entire sleep session) to live solely within a -quite large- retrieval region, surrounded by ergodicity: noticing that at this final stage of sleeping the network approached the Kanter-Sompolinsky model KanterSompo , it shines why these Authors called their model associative recall of memory without errors.

Acknowledgments

The Authors acknowledge partial financial fundings by MIUR, via FFABR2018-(Barra) and via Rete Match - Progetto Pythagoras (CUP:J48C17000250006) and by INFN.