Overlap matrix concentration in optimal Bayesian inference

TL;DR

This paper proves that in optimal Bayesian inference models with vector signals, the overlap matrix concentrates under perturbation, ensuring the validity of replica symmetric assumptions for various high-dimensional inference problems.

Contribution

It demonstrates the concentration of the overlap matrix in Bayesian models with finite-dimensional vector components, validating replica symmetry in the optimal setting.

Findings

Overlap matrix concentrates under perturbation.

Results apply to low-rank tensor factorization.

Validates replica symmetric assumptions in Bayesian inference.

Abstract

We consider models of Bayesian inference of signals with vectorial components of finite dimensionality. We show that, under a proper perturbation, these models are replica symmetric in the sense that the overlap matrix concentrates. The overlap matrix is the order parameter in these models and is directly related to error metrics such as minimum mean-square errors. Our proof is valid in the optimal Bayesian inference setting. This means that it relies on the assumption that the model and all its hyper-parameters are known so that the posterior distribution can be written exactly. Examples of important problems in high-dimensional inference and learning to which our results apply are low-rank tensor factorization, the committee machine neural network with a finite number of hidden neurons in the teacher-student scenario, or multi-layer versions of the generalized linear model.

Equations

$$X\sim P_0(\cdot |{\theta }_0).$$

$$\tilde{Y}\sim P_{\mathrm{out}}(\cdot |X,{\theta }_{\mathrm{out}}),$$

$$P(X=x|\tilde{Y},\theta )=P(x|\tilde{Y},\theta )$$

$$=\frac{1}{{\mathcal{Z}}_{0,n}(\tilde{Y},\theta )}{P}_0(x|{\theta }_0)\exp \{-{\mathcal{H}}_0(x,\tilde{Y},{\theta }_{\mathrm{out}})\}.$$

$${\mathcal{H}}_0(x,\tilde{Y},{\theta }_{\mathrm{out}})\equiv -\ln P_{\mathrm{out}}(\tilde{Y}|x,{\theta }_{\mathrm{out}})$$

$$f_{0,n}\equiv -\frac{1}{n}\mathbb{E}\ln {\mathcal{Z}}_{0,n}(\tilde{Y},\theta )=-\frac{1}{n}\mathbb{E}\ln \int d{P}_0(x|{\theta }_0)\exp \{-{\mathcal{H}}_0(x,\tilde{Y},{\theta }_{\mathrm{out}})\}.$$

$$\frac{1}{n}I(X;\tilde{Y}|\theta )=f_{0,n}-\frac{1}{n}H(\tilde{Y}|X,\theta ).$$

$$Q\equiv \frac{1}{n}{X}^{⊺}x=\frac{1}{n}\sum _{i=1}^n{X}_i{x}_i^{⊺},\text{or componentwise}{Q}_{k{k}^{\prime }}\equiv \frac{1}{n}\sum _{i=1}^n{X}_{ik}{x}_{i{k}^{\prime }}.$$

$$\displaystyle{\rm MMSE}\equiv\min_{\widehat{x}}\frac{1}{n}\mathbb{E}\big{[}\|X-\widehat{x}(\widetilde{Y},\theta)\|^{2}_{\rm F}\big{]}=\frac{1}{n}\mathbb{E}\big{[}\|X-\langle x\rangle_{0}\|^{2}_{\rm F}\big{]}=\mathbb{E}\big{[}\|X_{1}\|^{2}\big{]}-{\rm Tr}\,\mathbb{E}\langle Q\rangle_{0}$$

$$\displaystyle\frac{1}{n}\mathbb{E}\big{[}(X-\langle x\rangle_{0})^{\intercal}(X-\langle x\rangle_{0})\big{]}=\mathbb{E}\big{[}X_{1}^{\intercal}X_{1}\big{]}-\mathbb{E}\langle Q\rangle_{0}\,.$$

$$\displaystyle{\rm mMMSE}\equiv\frac{1}{n^{2}}\sum_{i,j=1}^{n}\mathbb{E}\big{[}(X_{i}^{\intercal}X_{j}-\langle x_{i}^{\intercal}x_{j}\rangle_{0})^{2}\big{]}=\mathbb{E}\big{[}(X_{1}^{\intercal}X_{2})^{2}\big{]}-\mathbb{E}\big{\langle}\|Q\|^{2}_{\rm F}\big{\rangle}_{0}+{\cal O}_{S}(1/n)\,.$$

$$\displaystyle\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}\Big{[}\Big{(}\sum_{k\in{\cal S}}X_{ik}-\Big{\langle}\sum_{k\in{\cal S}}x_{ik}\Big{\rangle}_{0}\Big{)}^{2}\Big{]}=\sum_{(k,k^{\prime})\in{\cal S}^{2}}\big{(}\mathbb{E}[X_{1k}X_{1k^{\prime}}]-\mathbb{E}\langle Q_{kk^{\prime}}\rangle_{0}\big{)}\,.$$

$${\tilde{Y}}_{i_1\dots i_p}=n^{\frac{1-p}{2}}\sum _{k=1}^K{X}_{i_{1}k}{X}_{i_{2}k}\dots X_{i_{p}k}+{\tilde{Z}}_{i_1\dots i_p},1\le i_1\le i_2\le \dots \le i_p\le n.$$

$$\displaystyle\widetilde{Y}_{\mu}\sim p_{\rm out}\Big{(}\,\cdot\,\Big{|}\sum_{i=1}^{n}\theta_{\mu i}X_{i}\Big{)},\qquad 1\leq\mu\leq m\,.$$

$${\tilde{Y}}_{\mu }=\text{ sign}\sum _{k=1}^K\text{ sign}\sum _{i=1}^n{\theta }_{\mu i}{X}_{ik},1\le \mu \le m.$$

$$\displaystyle\begin{cases}X^{(L)}_{i_{L}}\sim p_{\rm out}^{(L)}\big{(}\,\cdot\,\big{|}\sum_{j=1}^{n_{L-1}}\theta_{i_{L}j}^{(L)}X_{j}^{(L-1)}\big{)}\,,&1\leq i_{L}\leq n_{L}\,,\\ X_{i_{L-1}}^{(L-1)}\sim p_{\rm out}^{(L-1)}\big{(}\,\cdot\,\big{|}\sum_{j=1}^{n_{L-2}}\theta_{i_{L-1}j}^{(L-1)}X_{j}^{(L-2)}\big{)}\,,&1\leq i_{L-1}\leq n_{L-1}\,,\\ \qquad\qquad\qquad\qquad\vdots\\ X_{i_{1}}^{(1)}\sim p_{\rm out}^{(1)}\big{(}\,\cdot\,\big{|}\sum_{j=1}^{n_{0}}\theta_{i_{1}j}^{(1)}X_{j}^{(0)}\big{)}\,,&1\leq i_{1}\leq n_{1}\,.\end{cases}$$

$$\displaystyle\begin{cases}\widetilde{Y}_{ij}=n^{-1/2}\,\sum_{k=1}^{K}X_{ik}X_{jk}+\widetilde{Z}_{ij}\,,&1\leq i\leq j\leq n\,,\\ X_{i}\sim p_{\rm out}\big{(}\,\cdot\,\big{|}\sum_{j=1}^{n_{0}}\theta_{ij}X_{j}^{(0)}\big{)}\,,&1\leq i\leq n\,.\end{cases}$$

$$\displaystyle\mathbb{E}\big{\langle}g(Y,x^{(1)},\dots,x^{(k)})\big{\rangle}=\mathbb{E}\big{\langle}g(Y,X,x^{(2)},\dots,x^{(k)})\big{\rangle}\,.$$

$$Y=X{\lambda }_n^{1/2}+Z,\text{or componentwise}{Y}_i={\lambda }_n^{1/2}{X}_i+Z_i\text{for}1\le i\le n.$$

$${\lambda }_n\equiv s_n\tilde{\lambda },$$

$$\displaystyle{\cal D}_{K}\equiv\big{\{}\tilde{\lambda}\in\mathbb{R}^{K\times K}:\tilde{\lambda}_{kk^{\prime}}=\tilde{\lambda}_{k^{\prime}k}\in(1,2)\ \forall\ k\neq k^{\prime},\tilde{\lambda}_{kk}\in(2K,2K+1)\ \forall\ k\big{\}}\,.$$

$$\displaystyle{\cal D}_{n,K}\equiv\big{\{}\lambda\in\mathbb{R}^{K\times K}:\lambda_{kk^{\prime}}=\lambda_{k^{\prime}k}\in(s_{n},2s_{n})\ \forall\ k\neq k^{\prime},\lambda_{kk}\in(2Ks_{n},(2K+1)s_{n})\ \forall\ k\big{\}}\,.$$

$${\lambda }_n^{1/2}=\sqrt{s_n}{\tilde{\lambda }}^{1/2}.$$

$$\{\begin{array}{ll}\tilde{Y}& \sim P_{\mathrm{out}}(\cdot |X,{\theta }_{\mathrm{out}}),\\ Y_i& ={\lambda }_n^{1/2}{X}_i+Z_i,1\le i\le n.\end{array}$$

$$\displaystyle{\cal H}_{\lambda}(x,Y(X,Z),\lambda_{n})\equiv\sum_{i=1}^{n}\Big{(}\frac{1}{2}x_{i}^{\intercal}\lambda_{n}x_{i}-x_{i}^{\intercal}\lambda_{n}X_{i}-x_{i}^{\intercal}\lambda_{n}^{1/2}\,Z_{i}\Big{)}$$

$$P(x|\tilde{Y},Y,\theta ,{\lambda }_n)=\frac{1}{{\mathcal{Z}}_n(\tilde{Y},Y,\theta ,{\lambda }_n)}{P}_0(x|{\theta }_0)\exp \{-{\mathcal{H}}_0(x,\tilde{Y},{\theta }_{\mathrm{out}})-{\mathcal{H}}_{\lambda }(x,Y,{\lambda }_n)\}$$

$$⟨g⟩\equiv \int dP(x|\tilde{Y},Y,\theta ,{\lambda }_n)g(x)$$

$$f_n=f_n({\lambda }_n)$$

$$=-\frac{1}{n}\mathbb{E}\ln \int d{P}_0(x|{\theta }_0)\exp \{-{\mathcal{H}}_0(x,\tilde{Y},{\theta }_{\mathrm{out}})-{\mathcal{H}}_{\lambda }(x,Y,{\lambda }_n)\}$$

$$\displaystyle\mathbb{E}\Big{[}\Big{(}-\frac{1}{n}\ln{\cal Z}_{n}(\widetilde{Y},Y,\theta,\lambda_{n})-f_{n}(\lambda_{n})\Big{)}^{2}\Big{]}\leq\frac{C_{f}}{n}\,.$$

$$|f_{0,n}-f_n|\le S^2(2K+1)K^2{s}_n.$$

Full text

†† ​​​​​​​​​​​The Abdus Salam International Center for Theoretical Physics, Trieste, Italy.

jbarbier@ictp.it

1  Introduction

This decade is witnessing a burst of mathematical studies related to high-dimensional inference and learning problems. One reason is that an important arsenal of methods, developed in particular by the physicists and mathematicians working on the rigorous aspects of spin glasses, has found a new rich playground where it can be applied with success [1, 2, 3, 4, 5, 6, 7]. Models in learning like the perceptron and Hopfield neural networks have been analyzed in depth since the eighties by the physics community [8, 9, 10, 11, 12, 13], or in inference, e.g., in the context of communications and error correcting codes [14, 15], using powerful but non-rigorous techniques such as the replica and cavity methods [16, 17]. But due to the difficulty and richness of these models rigorous results experienced some delay with respect to (w.r.t.) the physics appoaches and were restricted to very specific models such as the famous Sherrington-Kirkpatrick model [2, 6, 7]. The trend is changing and it is fair to say that the gap between heuristic (yet often exact) physics approaches and rigorous ones is quickly shrinking. In particular important progress towards the vindication of the replica and cavity methods has been made recently in the context of high-dimensional Bayesian inference and learning. Examples of problems in this class where the physics approaches are now rigorously settled include low-rank matrix and tensor factorization [18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32], random linear and generalized estimation [33, 34, 35, 36, 37, 38], models of neural networks in the teacher-student scenario [37, 39, 40], or sparse graphical models such as error-correcting codes and block models [41, 42, 43].

All these results are based in some way or another on the control of the fluctuations of the order parameter of the problem, the overlap, which quantifies the quality of inference. Optimal Bayesian inference –optimal meaning that the true posterior is known– is an ubiquitous setting in the sense that the overlap can be shown to concentrate, and this in the whole regime of parameters (amplitude of the noise, number of observations/data points divided by the number of parameters to infer etc). When the overlap is self-averaging (which is the case in optimal Bayesian inference under a proper perturbation, see Theorems 4 and 5) then one expects replica symmetric variational formulas for the asymptotic free energy or mutual information density, as understood a long time ago by physicists [44, 45]. Actually in the physics literature replica symmetry is generally the term used to precisely mean that the order parameter concentrates. This is in contrast with models where the overlap is not self-averaging, like in spin glasses at low temperature or combinatorial optimization problems, which leads to more complicated formulas for the free energy computed using Parisi’s replica symmetry breaking scheme [46, 16, 17, 5, 6, 7].

In most of the studied statistical models the overlap order parameter is a scalar. In the context of optimal Bayesian inference it is now quite standard to show that when the overlap is a scalar it is self-averaging in the whole phase diagram, see, e.g., [47, 28]. The techniques to do so have been developed in the context of communications starting with [48, 49, 33] (and then generalized in [50, 41]), and are extensions of methods used in the analysis of spin glasses [1, 51, 52, 53, 54, 4, 5, 55]. In this paper we consider instead Bayesian inference problems where the signal to be reconstructed is made of vectorial components. In this case the overlap is a matrix and the associated replica formulas are variational formulas over matrices. The concentration techniques developed for scalar overlaps do not apply directly, and need to be extended using new non-trivial ideas. In particular, new difficulties will appear w.r.t. the scalar case due to the fact that overlap matrices are not symmetric objects. Examples of problems where matrix overlaps appear are the factorization of matrices and tensors of rank greater than one [24], or the so-called committee machine neural network with few hidden neurons [56, 57, 58, 39]. In the context of spin glasses, matrix overlap order parameters have also appeared recently in studies of vectorial versions of the Potts and mixed $p$-spin models by Panchenko [59, 60]; in these models replica symmetry breaking occurs and the overlap does not concentrate. Let us also mention the recent work by Agliari and co-workers [61] on a “multi-species” version of the Hopfield model, where a matrix order parameter also appears. There concentration of overlap, in the replica symmetric region where concentration is expected, is assumed based on strong physical arguments. In the context of optimal Bayesian inference the situation is more favorable than in spin glasses: thanks to special identities that follow from Bayes’ rule and known as “Nishimori identities” in statistical physics (see, e.g., [62, 63]), we show in this paper how to control the overlap fluctuations in the whole phase diagram111Let us mention another particular setting where the (scalar) overlap as well as its multi-body generalizations (which appear in diluted problems, i.e., problems defined by sparse graphical models) can be controlled under proper perturbations for all the parameters values in the phase diagram: ferromagnetic models [64]..

Section 2 presents the general setting, gives a few examples of models covered by our results, and explains the important Nishimori identity for optimal Bayesian inference problems. In section 3 we introduce the perturbation needed in order to prove overlap concentration, and then give our main results Theorems 4 and 5. Then in section 4 we provide the proof of Theorem 5. Finally in section 5 we prove an important intermediate concentration result for another matrix, that will be key in controlling the overlap.

2  Optimal Bayesian inference of signals with vector entries

2.1 Setting

Consider a model where a signal $X=(X_{ik})\in[-S,S]^{n\times K}$ made of $n$ components (indexed by $i$), that are each a $K$-dimensional bounded real vector (with dimensions indexed by $k$), is generated probabilistically. Its probability distribution $P_{0}$, called prior, may depend on a generic hyper-parameter $\theta_{0}\in\Theta_{0}$ with $\Theta_{0}$ an arbitrary real set, i.e.,

[TABLE]

We assume that the prior has bounded support (with $S<+\infty$ arbitrarily large but independent of $n$). Then some real data (also called observations) $\widetilde{Y}$ are generated conditionally on the unknown signal $X$ and an hyper-parameter $\theta_{\rm out}$ belonging to a generic real set $\Theta_{\rm out}$. Namely, the data

[TABLE]

with $\widetilde{Y}\in\widetilde{\cal Y}$ a generic real set: the data $\widetilde{Y}$ and hyper-parameters $\theta_{0}$, $\theta_{\rm out}$ can be real numbers, vectors, tensors etc. The conditional distribution $P_{\rm out}$ is called likelihood, or “output channel”. We also assume that the hyper-parameters $\theta_{0}$ and $\theta_{\rm out}$ are also probabilistic, with respective probability distributions $P_{\theta_{0}}$ supported on $\Theta_{0}$, and $P_{\theta_{\rm out}}$ supported on $\Theta_{\rm out}$. This formulation includes the case of deterministic hyper-parameters choosing Dirac delta measures $P_{\theta_{\rm out}}=\delta_{\theta_{\rm out}}$ and $P_{\theta_{0}}=\delta_{\theta_{0}}$.

The inference task is to recover the signal $X$ as accurately as possible given the data $\widetilde{Y}$. We moreover assume that the hyper-parameters $\theta\equiv(\theta_{0},\theta_{\rm out})$, the likelihood $P_{\rm out}$ and the prior $P_{0}$ are known to the statistician, and call this setting optimal Bayesian inference.

The information-theoretical optimal way of reconstructing the signal follows from its posterior distribution. Using Bayes’ formula the posterior reads

[TABLE]

Employing the language of statistical mechanics we call

[TABLE]

the base Hamiltonian, while the posterior normalization ${\cal Z}_{0,n}(\widetilde{Y},\theta)$ is the partition function of the base inference model. Finally the averaged free energy is minus the averaged log-partition function:

[TABLE]

The average $\mathbb{E}=\mathbb{E}_{\theta}\mathbb{E}_{X|\theta_{0}}\mathbb{E}_{\widetilde{Y}|X,\theta_{\rm out}}$ is over the randomness of $(\theta,X,\widetilde{Y})$. These are jointly called the quenched variables as they are fixed by the realization of the problem, in contrast with the dynamical variable $x$ which fluctuates according to the posterior. In general $\mathbb{E}$ will be used for an average w.r.t. all random variables in the ensuing expression. Note that the averaged free energy is nothing else than the Shannon entropy density of the observations (given the hyper-parameters): $f_{0,n}=\frac{1}{n}H(\widetilde{Y}|\theta)$. Therefore it is simply related to the mutual information density between the observations and the signal:

[TABLE]

The conditional entropy $\frac{1}{n}H(\widetilde{Y}|X,\theta)$ is often easy to compute, as opposed to the averaged free energy.

We call model (2.1) the “base model” in contrast with the perturbed model presented in section 3, a slightly modified version of the base model where additional side-information is given, and for which overlap concentration can be proved without altering the thermodynamic $n\to+\infty$ limit of the averaged free energy (if it exists), see Lemma 3.

The central object of interest is the $K\times K$ overlap matrix (or simply overlap) $Q=(Q_{kk^{\prime}})$ defined as

[TABLE]

Here $x$ is a sample drawn according to the posterior distribution and $X$ is the signal (all vectors are columns, including isolated rows of matrices, and transposed vectors are rows). The overlap contains a lot of information. E.g., the minimum mean-square error (MMSE), an error metric often considered in signal processing, is related to it through

[TABLE]

where we denote $\langle-\rangle_{0}$ the expectation w.r.t. the posterior (2.1) of the base model. The minimization is over all functions of $(\widetilde{Y},\theta)$ in $\mathbb{R}^{n\times K}$, $\|-\|_{\rm F}$ is the Frobenius norm, $X_{i}=(X_{ik})_{k}\in[-S,S]^{K}$ is the $i$-th row of $X$. A simple fact from Bayesian inference is that the estimator minimizing the MMSE is the posterior mean $\langle x\rangle_{0}\equiv\mathbb{E}[X|\widetilde{Y},\theta]$. One may also be interested in the $K\times K$ MMSE matrix, which provides information about the individual dimensions in the row space of $X$:

[TABLE]

This can be important in settings where some dimensions can be recovered while others cannot (see [65] and references therein) or, e.g., to study the “specialization phase transition” of the neurons during learning in some models of neural networks [39]. The usual scalar MMSE (2.2) is just the trace of this richer object.

Another metric of interest in problems where, e.g., the sign of the signal is lost due to symmetries is the matrix-MMSE (not to be confused with the MMSE matrix above). Again, it is related to the overlap (the notation $A=B+{\cal O}_{S}(1/n)$ means $|A-B|\leq C(S)/n$ for some positive constant $C(S)$ depending only on the prior support $S$):

[TABLE]

Finally if one is interested in estimating the sum over a subset ${\cal S}\subseteq\{1,\ldots,K\}$ of the the signal entries a possible error metric is

[TABLE]

2.2 Examples

Let us provide some examples of models that fall under the setting of optimal Bayesian inference with vector variables as described in the previous section.

In the symmetric order-$p$ rank-$K$ tensor factorization problem, the data-tensor $\widetilde{Y}=(\widetilde{Y}_{i_{1}\ldots i_{p}})$ is generated through the observation model

[TABLE]

Here $\widetilde{Z}$ is a Gaussian noise tensor with independent and identically distributed (i.i.d.) ${\cal N}(0,1)$ entries for $1\leq i_{1}\leq i_{2}\leq\ldots\leq i_{p}\leq n$, and the signal components are i.i.d., i.e., with a prior of the form $P_{0}=p_{0}^{\otimes n}$ with $p_{0}$ a probability distribution supported on $[-S,S]^{K}$. The case $p=2$ is known as the Wigner spike model, or low-rank matrix factorization, and is one of the simplest probabilistic model for principal component analysis. In both the analysis of [31, 32] the matrix overlap concentration is a key result. The Wigner spike model is an example of model where the signal’s sign is lost, and therefore a relevant error metric is the matrix-MMSE (2.3).

Another model is the following generalized linear model (GLM) (recall $X_{i}\in[-S,S]^{K}$):

[TABLE]

Note that here the $m$ observations are i.i.d. given $\mathbb{R}^{m\times n}\ni\theta_{\rm out}=(\theta_{\mu})_{\mu=1}^{m}$ and $X$; this is the reason for the notation $p_{\rm out}$ instead of $P_{\rm out}$, the latter representing the full likelihood while the former is the conditional distribution of a single data point, i.e., $P_{\rm out}(\,\cdot\,|\theta_{\rm out}X)=\otimes_{\mu=1}^{m}p_{\rm out}(\,\cdot\,|X^{\intercal}\theta_{\mu})$. We also assume that the prior $P_{0}=p_{0}^{\otimes n}$ is decoupled over the $n$ signal components and $m=\Theta(n)$. A particular simple deterministic case is

[TABLE]

This model is the committee machine mentionned in the introduction [37, 39]. Here $(X_{ik})_{i=1}^{n}$ can be interpreted as the weights of the $k$-th hidden neuron, and $(\theta_{\mu})$ are $n$-dimensional data points used to generate the labels $(\widetilde{Y}_{\mu})$. The teacher-student scenario in which our results apply corresponds to the following: the teacher network (2.6) (or (2.5) in general) generates $\widetilde{Y}$ from the data $\theta_{\rm out}$. The pairs $(\widetilde{Y}_{\mu},\theta_{\mu})$ are then used in order to train (i.e., learn the weights of) a student network with exactly the same architecture.

A richer example is a multi-layer version of the GLM above:

[TABLE]

with an input $X^{(0)}\sim P_{0}$ factorized as $P_{0}=p_{0}^{\otimes n_{0}}$. In this model $(X^{(\ell)})_{\ell=1}^{L-1}$ represent intermediate hidden variables, the visible variable $X^{(L)}=\widetilde{Y}$ is the data, and $\theta_{\rm out}=(\theta^{(\ell)})$ with $\theta^{(\ell)}$ representing the weight matrix at the $\ell$-th layer. Note that in the single layer version (2.5), $\theta_{\rm out}$ was instead interpreted as data points and $X$ was the weight vector to learn/infer. Also $n^{(\ell)}=\Theta(n_{0})$ for $\ell=1,\ldots,L$. This scaling for the variables sizes is often assumed in order not to make the inference of $X^{(0)}$ from $X^{(L)}$ impossible, nor trivial. This multi-layer GLM has been studied by various authors for the $K=1$ case and when the output components $X_{j}^{(\ell)}$ are scalars [66, 67, 68, 40, 69]. But one can define generalizations where these are multi-dimensional, in which case overlap matrices naturally arise.

A final example could be another combination of complex statistical models such as, e.g., the following symmetric matrix factorization problem where the hidden low-rank representation $X$ of the matrix is itself generated from a generalized linear model over a more primitive signal $X^{(0)}$:

[TABLE]

Here again some factorization structure for the prior $P_{0}=p_{0}^{\otimes n_{0}}$ of $X^{(0)}$ may be assumed, and $n=\Theta(n_{0})$. Such model has recently been studied in [70].

2.3 The Nishimori identity

The following identity is a simple consequence of Bayes’ formula, and applies to optimal Bayesian inference.

Lemma 1** (Nishimori identity).**

Let $(X,Y)$ be a couple of random variables with joint distribution $P(X,Y)$ and conditional distribution $P(X|Y)$. Let $k\geq 1$ and let $x^{(1)},\dots,x^{(k)}$ be i.i.d. samples from the conditional distribution $P(\,\cdot\,|Y)$. These are called “replicas”. Let us denote $\langle-\rangle$ the expectation operator w.r.t. the product conditional distribution $P(x^{(1)}|Y)P(x^{(2)}|Y)\dots P(x^{(k)}|Y)$ acting on the replicas, and $\mathbb{E}$ the expectation w.r.t. the joint distribution $P(X,Y)$. Then, for any continuous bounded function $g$,

[TABLE]

Proof.

It is equivalent to sample the couple $(X,Y)$ according to its joint distribution or to sample first $Y$ according to its marginal distribution and then to sample $X$ conditionally on $Y$ from the conditional distribution. Thus the two $(k+1)$-tuples $(Y,x^{(1)},\dots,x^{(k)})$ and $(Y,X,x^{(2)},\dots,x^{(k)})$ have the same law. ∎

In practice the Nishimori identity222This identity has been abusively called “Nishimori identity” in the statistical physics literature despite that it is a simple consequence of Bayes’ formula. The “true” Nishimori identity concerns models with one extra feature, namely a gauge symmetry which allows to eliminate the input signal, and the expectation over the signal $X$ in expressions of the form $\mathbb{E}\langle-\rangle$ can therefore be dropped. allows to “replace” the ground-truth signal $X$ by an independent replica, and vice-versa, in expressions involving only other replicas and the observations. Again, by replicas we mean conditionally independent samples drawn according to the posterior.

3  The vectorial Gaussian channel perturbation

In order to “force” the overlap to concentrate we need to have access to infinitesimal side-information $Y$ in addition to the observations $\widetilde{Y}$. This side information is coming from the following vectorial Gaussian channel:

[TABLE]

Here the signal $X$ is the same as in the base inference model. The observations $(Y_{i})$, the signal components $(X_{i})$ and i.i.d. Gaussian noise variables $(Z_{i})\sim{\cal N}(0,I_{K})^{\otimes n}$ are all $K$-dimensional vectors. The signal-to-noise (SNR) matrix controlling the signal strength

[TABLE]

with a positive sequence $(s_{n})\in(0,1]^{\mathbb{N}}$ that tends to $0_{+}$ slowly enough (the rate will be specified later), and $\tilde{\lambda}$ belongs to ${\cal D}_{K}$ defined as

[TABLE]

Therefore, $\lambda_{n}$ belongs to the set

[TABLE]

Matrices belonging to ${\cal D}_{n,K}$ are symmetric strictly diagonally dominant with positive entries (of the order of $s_{n}$) and thus ${\cal D}_{n,K}\subset{\cal S}_{K}^{+}$, where ${\cal S}_{K}^{+}$ is the set of symmetric positive definite matrices of dimension $K\times K$, see [71]. As $\lambda_{n}\in{\cal D}_{n,K}$ it possesses a unique principal square root matrix denoted

[TABLE]

The advantage of working with the ensemble ${\cal D}_{n,K}$ is the following. We require that the SNR matrix $\lambda_{n}$ always belong to ${\cal S}_{K}^{+}$ so that its square root is real and unique. For a generic positive matrix in ${\cal S}_{K}^{+}$, but not necessarily in ${\cal D}_{n,K}$, one cannot vary its (symmetric) elements independently because doing so the matrix might not be positive definite anymore; the constraint $\lambda_{n}\in{\cal S}_{K}^{+}$ is a “global” constraint over the matrix elements. In contrast if $\lambda_{n}\in{\cal D}_{n,K}$ we can vary its elements independently (as long as it remains in ${\cal D}_{n,K}$) without the possibility that $\lambda_{n}$ falls out of ${\cal S}_{K}^{+}$.

The perturbed inference model is then the following obervation model:

[TABLE]

It is called “perturbed model” because the original observation model has been slightly modified by adding new observations coming from (3.1) that are “weak” (as $s_{n}\to 0_{+}$). The perturbation Hamiltonian associated with the observation channel (3.1) is

[TABLE]

using the symmetry of the SNR matrix. The total Hamiltonian is therefore the sum of the base Hamiltonian and the perturbation one. The posterior of the perturbed model, written in the standard Gibbs-Boltzmann form of statistical mechanics, is

[TABLE]

where again the partition function ${\cal Z}_{n}(\widetilde{Y},Y,\theta,\lambda_{n})$ is simply the normalization constant. We also define the Gibbs-bracket $\langle-\rangle$ as the expectation operator w.r.t. the posterior of the perturbed model:

[TABLE]

for any function $g$ s.t. its expectation exists. Thus $\langle g\rangle$ depends on the quenched variables $(\widetilde{Y},Y,\theta)$ and the perturbation parameter $\lambda_{n}$.

It is crucial to notice the following. The perturbation is constructed from an inference channel (3.1) which form is known (i.e., it is known that the noise is a realization of ${\cal N}(0,I_{n})$ and the signal-to-noise ratio matrix $\lambda_{n}$ is given). Therefore the perturbed model (3.3) is a proper inference problem in the optimal Bayesian inference setting. This means that in addition to the data $(\widetilde{Y},Y)$, the statistician fully knows the data generating model, namely the likelihood $P_{\rm out}$ and the additive Gaussian nature of the noise in the second channel in (3.3), the prior $P_{0}$ as well as all hyper-parameters $(\theta,\lambda_{n})$, and is therefore able to write the true posterior (3.5) of the model when estimating the signal. As a consequence the Nishimori identity Lemma 1 applies to the perturbed model and its bracket $\langle-\rangle$.

An important quantity is the averaged free energy of the perturbed model:

[TABLE]

where the above expectation $\mathbb{E}\equiv\mathbb{E}_{\theta}\mathbb{E}_{X|\theta_{0}}\mathbb{E}_{\widetilde{Y}|X,\theta_{\rm out}}\mathbb{E}_{Y|X,\lambda_{n}}$ carries over the random hyper-parameters, the ground-truth signal (given $\theta_{0}$) and the data generated according to (3.3), but not over $\lambda_{n}$ which remains fixed. Later we will average quantities w.r.t. $\lambda_{n}\in{\cal D}_{n,K}$, but in this case we will explicitely write $\mathbb{E}_{\lambda}$.

In order to prove the concentration of the overlap we need the following crucial hypothesis:

Hypothesis 2** (Free energy concentration).**

The free energy (3.7) of the perturbed model concentrates at the optimal rate, namely there exists a constant $C_{f}=C_{f}(K,P_{0},P_{\rm out},P_{\theta_{0}},P_{\theta_{\rm out}})$ that may depend on everything but $n$, and s.t.

[TABLE]

There are some remarks to be made here. The first one is related to the scenarios where this hypothesis can be verified. For purely generic optimal inference models without any restricting assumptions on the form of the distributions $(P_{0},P_{\rm out},P_{\theta_{0}},P_{\theta_{\rm out}})$ it is generally very hard, if not wrong, to try proving (3.8). The model must be “random enough” and possess some underlying factorization structure for such hypothesis to be true. The most studied case in the literature is when the prior and the likelihood factorize, namely $P_{0}=p_{0}^{\otimes n}$ and the data points are i.i.d. given $(X,\theta_{\rm out})$. The examples (2.4)–(2.6) fall in this class. Under such independence/factorization assumptions it is quite straightforward to prove that the free energy concentrates using standard techniques333In the context of statistical physics of spin glasses, these factorization and independence properties translate into the fact that the external fields $(h_{i})$ act independently on each spins, and the coupling constants $(J_{ij})$ act pairwise (or on a finite subsets of variables for $p$-spin types of models), and are independently drawn from some distribution. Therefore, in this context, free energy concentration is standard to prove. This is not necessarily the case in inference with generic prior and noise models, that can induce correlations preventing self-averaging. This is the reason why free energy concentration is here stated as an hypothesis. (see, e.g., [28, 37]). But such simple factorization properties are not always there, as illustrated by examples (2.7), (2.8). In these two last examples it is a perfectly valid question to wonder whether the overlap of the hidden variables do concentrate444Note that proving concentration of the overlap for a hidden variable requires a perturbation of the form (3.1) over the hidden variable, not over $X^{(0)}$, which in this case is just interpreted as a constitutive element of the prior of the hidden variable of interest, see [40] where this is done. (this question is crucial in the analysis of [40]). The hidden variables have very complex structured prior (i.e., probability distribution), with highly non-trivial factorization properties, in which case proving (3.8) requires work. See, e.g., [40] where this has been done for the multi-layer GLM (2.7) with a single hidden layer ($L=2$) where this is already challenging.

The second remark is that the perturbation does not change the limit of the averaged free energy:

Lemma 3** (The base and perturbed models have same asymptotic averaged free energy).**

We have

[TABLE]

*Therefore $f_{0,n}$ and $f_{n}$ have same thermodynamic limit, provided it exists.

Proof.

It follows from identities (4.8), (4.9) in section 4.1 that $\|\nabla_{\lambda_{n}}f_{n}\|_{\rm F}\leq\|\mathbb{E}\langle Q\rangle\|_{\rm F}$. By the mean value theorem $|f_{0,n}-f_{n}|\leq\|\nabla_{\lambda_{n}}f_{n}\|_{\rm F}\|\lambda_{n}\|_{\rm F}$ and thus $|f_{0,n}-f_{n}|\leq\|\mathbb{E}\langle Q\rangle\|_{\rm F}\|\lambda_{n}\|_{\rm F}$. By definition matrices in ${\cal D}_{n,K}$ have positive entries bounded by $(2K+1)s_{n}$. Therefore as $\lambda_{n}\in{\cal D}_{n,K}$ so $\|\lambda_{n}\|_{\rm F}\leq(2K+1)Ks_{n}$. By hypothesis the prior support is contained in $[-S,S]$ so we have $\|\mathbb{E}\langle Q\rangle\|_{\rm F}\leq S^{2}K$, and thus the result. ∎

3.1 Main results

All along this paper we denote $C(U)$ a generic positive numerical constant depending only on the parameters $U$. E.g., $C(C_{f},K,S)>0$ depends only on $C_{f}$ appearing in (3.8), the variables dimensionality $K$ and on the prior support $S$. Let us denote the average over the matrix $\lambda_{n}\in{\cal D}_{n,K}$ appearing in the perturbation (3.1) as

[TABLE]

Here ${\rm Vol}({\cal D}_{n,K})$ is the volume of ${\cal D}_{n,K}$ which vanishes as $n\to+\infty$ (there are $K(K+1)/2$ independent entries in $\lambda_{n}$ as it is symmetric). Recall the notation $\langle-\rangle$ for the expectation w.r.t. the posterior of the perturbed inference model (3.6).

In order to give our first result we need to introduce the overlap between two replicas

[TABLE]

where, again, replicas are conditionally i.i.d. random variables drawn accroding to the posterior (3.5) of the perturbed model (and thus share the same quenched variables): $(x^{(1)},x^{(2)})\sim P(\,\cdot\,|\widetilde{Y},Y,\theta,\lambda_{n})^{\otimes 2}$. By a slight abuse of notation let us continue to use the same bracket notation for the expectation of functions of replicas w.r.t. to the product posterior measure:

[TABLE]

Our main results are the following concentration theorems for the overlap in a (perturbed) model of optimal Bayesian inference. We start with the first type of fluctuations, namely the fluctuations of the overlap w.r.t. the posterior distribution, or what is called “thermal fluctuations” in statistical mechanics. Note that for controlling these fluctuations we do not need that the free energy concentrates, i.e., the hypothesis (3.8) is not required. As a consequence this result is valid even for very complex models without any factorization properties for the signal’s prior nor for the likelihood (as long as they are defined in the optimal Bayesian setting). This result is a consequence of the precense of the perturbation combined with the Nishimori identity.

Theorem 4** (Thermal fluctuations of $Q$).**

Assume that the perturbed inference model is s.t. the Nishimori identity Lemma 1 holds. Let $(s_{n})\in(0,1]^{\mathbb{N}}$ a positive sequence verifying $s_{n}\to 0_{+}$ and $s_{n}n\to+\infty$. There exists positive constants $C(K,S)$ s.t.

[TABLE]

The next, stronger, result takes care of the additional fluctuations due to the quenched randomness, and requires the free energy concentration hypothesis:

Theorem 5** (Total fluctuations of $Q$).**

Assume that the perturbed inference model is s.t. $i)$ its free energy concentrates as in identity (3.8); $ii)$ the Nishimori identity Lemma 1 holds. Let $(s_{n})\in(0,1]^{\mathbb{N}}$ a positive sequence verifying $s_{n}\to 0_{+}$ and $s_{n}^{4}n\to+\infty$. There exists a positive constant $C(C_{f},K,S)$ s.t.

[TABLE]

Before entering the proof let us make a very last remark. There are problems with multiple overlaps. For example one may also consider the non-symmetric version of the tensor factorization problem. In this case $p$ matrices $X^{[p]}\in\mathbb{R}^{n_{p}\times K}$, with $n_{p}=\Theta(n)$ and with a possibly matrix-dependent prior $P_{0}^{[p]}$, are to be reconstructed from a data-tensor of the form

[TABLE]

In this case there is one overlap per matrix-signal to be inferred:

[TABLE]

It should be clear to the reader that all the setting described in this paper can be straightforwardly extended to include this case: one has to consider one perturbation channel of the form (3.1) per variable to be reconstructed (i.e., per matrix in the non-symmetric tensor factorization problem), each with its own independent matrix SNR:

[TABLE]

Then the total Hamiltonian is the sum of the base one and the $p$ perturbation Hamiltonians, and so forth.

4  Proof of concentration of the overlap matrix

For the sake of readibility we now drop the $n$ index in the matrix SNR:

[TABLE]

We use $l$, $l^{\prime}$ and $k$, $k^{\prime}$ for the variables dimension indices which are running from $1$ to $K$, and $i$, $j$ for the variables indices running from $1$ to $n$. When we write $l$ and $l^{\prime}$ we always implicitly mean $l^{\prime}\neq l$.

Let us start with some preliminary computations.

4.1 Preliminaries: properties of the matrix ${\cal L}$

The proof that the overlap concentrates relies on the concentration of another matrix defined as

[TABLE]

where we used

[TABLE]

The fluctuations of this matrix are easier to control than the ones of the overlap. This comes from the fact that ${\cal L}$ is related to derivatives of the free energy, which is self-averaging by hypothesis (3.8). First consider $l^{\prime}\neq l$. We have

[TABLE]

We used the Nishimori identity Lemma 1 which in this case implies

[TABLE]

Each time we use an identity that is a consequence of Lemma 1 we write a $\rm N$ on top of the equality (that stands for Nishimori). We integrate by part the Gaussian noise thanks to the formula

[TABLE]

and any bounded function $g$. This leads to

[TABLE]

We used that the derivative of the Hamiltonian (3.4) is

[TABLE]

We now exploit the symmetry of the matrices $x_{i}x_{i}^{\intercal}$ and $\langle x_{i}\rangle\langle x_{i}\rangle^{\intercal}$ in order to symmetrize the terms in (4.4) and then use the formula

[TABLE]

Identity (4.4) then becomes

[TABLE]

Similarly wo obtain for the diagonal terms

[TABLE]

Using this (4.1) becomes

[TABLE]

Therefore the expectation of ${\cal L}$ is directly related to the one of $Q$. It is thus natural to guess that if $\cal L$ concentrates onto its mean, the overlap should concentrate too. Indeed, the following concentration identity for ${\cal L}$ is key in proving Theorems 4 and 5. Note that the following proposition does not require the Nishimori identity (i.e., to be in the optimal Bayesian setting). But the Nishimori identity will be crucial when linking the fluctuations of $\cal L$ to those of $Q$.

Proposition 6** (Concentration of ${\cal L}$).**

Let $(s_{n})\in(0,1]^{\mathbb{N}}$ a positive sequence verifying $s_{n}\to 0_{+}$ and $s_{n}n\to+\infty$. There exists a positive constant $C(S,K)$ s.t.

[TABLE]

Moreover if $s_{n}^{4}n\to+\infty$ and the free energy concentrates as in identity (3.8), then there exists a constant $C(C_{f},K,S)$ s.t.

[TABLE]

Let us assume this result at the moment and show how it implies concentration of $Q$. We will then prove Proposition 6 later in section 5.

4.2 Thermal fluctuations: proof of Theorem 4

Let us start with the control of the fluctuations due to the posterior distribution.

Proof of (3.10) in Theorem 4.

By definition of the overlap we have

[TABLE]

using the Cauchy-Schwarz inequality. Note that the first term on the r.h.s. of this inequality is bounded by $C(S)$. We show next, using the Nishimori identity, that

[TABLE]

Thus we obtain, for large enough constants $C(K,S)$ and as $s_{n}n\to+\infty$,

[TABLE]

The concentration identity (4.10) in Proposition 6 then implies (3.10).

It remains to prove the crucial identity (4.13). Acting with the operator $\frac{1}{n}\frac{d}{d{\lambda_{ll}}}$ on both sides of (4.9), i.e., starting from the identity

[TABLE]

we obtain

[TABLE]

Computing the derivative of $\mathcal{L}_{ll}$ and using

[TABLE]

which follows from (4.6) we find, using (4.2) and similar computations as (4.4)–(4.7), that

[TABLE]

We used for the last step that the entries of the matrix \big{(}\frac{d\lambda^{1/2}}{d\lambda_{ll}}\big{)}^{2} are ${\cal O}(s_{n}^{-1})$. Indeed recall that

[TABLE]

where $\tilde{\lambda}^{1/2}$ is independent of $n$. Therefore for any $(l,l^{\prime})$

[TABLE]

where, by a slight abuse of notation, we mean here that each element of this matrix is ${\cal O}(s_{n}^{-1/2})$. Let us compute the following term appearing in (4.14):

[TABLE]

We need to simplify

[TABLE]

Using similar manipulations as for obtaining (4.4)–(4.7), i.e., by symmetrizing when possible in order to use (4.6), we simplify the first term in $T$:

[TABLE]

Similarly the second term in $T$ is

[TABLE]

Therefore, using again (4.6), $T$ is the equal to

[TABLE]

Plugging this expression in (4.17) and then simplifying using the Nishimori identity we obtain

[TABLE]

Together with (4.15) and (4.14) it ends the proof of (4.13), and therefore of (3.10) too. ∎

Proof of (3.11) in Theorem 4.

We denote the overlap between the replica $x=x^{(1)}$ and the ground-truth signal $X=x^{(0)}$ equivalently as

[TABLE]

and recall definition (3.9) for the overlap between two replicas $x^{(1)}$ and $x^{(2)}$. Recall also that we still use the bracket notation $\langle-\rangle$ for the expectation w.r.t. $P(x^{(1)}|\widetilde{Y},Y,\theta,\lambda_{n})P(x^{(2)}|\widetilde{Y},Y,\theta,\lambda_{n})$, the product posterior measure acting of the conditionally independent replicas.

The proof relies on the following relation which is a simple consequence of the Nishimori identity:

[TABLE]

The first fluctuations in (4.19), once averaged over $\lambda$, are controlled by (3.10) that we have just proven. The second fluctuations are controlled as follows:

[TABLE]

We recognize a similar form as (4.12). The derivation is then the same as the one of (3.10) based on (4.13) and yields

[TABLE]

This ends the proof of (3.11), and thus of Proposition 4. ∎

4.3 Total fluctuations: proof of Theorem 5

Now that we control the thermal fluctuations we are in position to prove our second main Theorem 5. It shows that if the free energy concentrates then the overlap not only concentrates w.r.t. the posterior distribution, but also w.r.t. the quenched variables. The spirit of the proof is similar to the derivation of the Ghirlanda-Guerra identities in the context of spin glasses555What we mean here is that, as for the proof of the Ghirlanda-Guerra identities, the present method is based on testing the overlap matrix $Q$ against the fluctuations of the derivative of the Hamiltonian (the ${\cal L}$ matrix given by (4.1), (4.2)). In the context of inference, related to the Nishimori line in spin glasses [62, 63], the derived identities really are a special case of the Ghirlanda-Guerra identities. [1]. Our goal here is to compute

[TABLE]

using that

[TABLE]

because of (4.8), (4.9), and that $\mathbb{E}\langle Q\rangle$ is symmetric. The crux of the proof is that from the quantity $\mathbb{E}_{\lambda}{\rm Tr}\,\mathbb{E}\langle Q({\cal L}-\mathbb{E}\langle{\cal L}\rangle)\rangle$ will appear the fluctuations of the overlap, and this quantity is small by Proposition 6 (with Cauchy-Schwarz).

It thus only remains to compute $\mathbb{E}_{\lambda}{\rm Tr}\,\mathbb{E}\langle Q{\cal L}\rangle$. Let us first consider the off-diagonal terms:

[TABLE]

We need to simplify the last term. Using (4.5),

[TABLE]

The first term of the r.h.s. of (4.22) has been simplified because $x_{i}x_{i}^{\intercal}$ is symmetric, allowing the symmetrization of $\frac{d\lambda^{1/2}}{d\lambda_{ll^{\prime}}}$ followed by the use of (4.6). In contrast the second term above lacks symmetry as the matrix $\langle x_{i}\rangle\langle Q_{ll^{\prime}}x_{i}^{\intercal}\rangle$ is not symmetric. This prevents the use of the mechanism employed for the first term. In order to face this difficulty we exploit the concentration of the overlap w.r.t. the posterior that has been shown previously. We can write

[TABLE]

by relation (3.10) in Proposition 4 (which relies on the Nishimori identity) and Cauchy-Schwarz, because the entries of the matrix $\frac{d\lambda^{1/2}}{d\lambda_{ll^{\prime}}}\lambda^{1/2}$ are bounded (recall that $\lambda^{1/2}\equiv\sqrt{s_{n}}\,\tilde{\lambda}^{1/2}$ where $\tilde{\lambda}^{1/2}$ is independent of $n$ with bounded entries, and (4.16)) as well as the support of the prior (so $|\langle x_{ik}\rangle|\leq S$). Now we can exploit symmetry and therefore write

[TABLE]

Combining everything in (4.22) and (4.21) yields

[TABLE]

using

[TABLE]

the latter matrix being symmetric. The notation $A=B+{\cal O}_{K,S}((s_{n}n)^{-1/4})$ we introduced means simply that $|A-B|\leq C(K,S)(s_{n}n)^{-1/4}$. We now consider the diagonal terms. Similarly

[TABLE]

Summing everything we obtain (note that we combine all the on and off-diagonal terms computed above inside a single double sum $\sum_{l,l^{\prime}}$, and we therefore need to remove the diagonal terms counted twice)

[TABLE]

Therefore, plugging this in (4.20) leads to

[TABLE]

A direct application of (3.11) in Proposition 4 together with Cauchy-Schwarz gives:

[TABLE]

as the overlap norm is bounded by $KS^{2}$. Similarly

[TABLE]

Therefore

[TABLE]

Finally by Cauchy-Schwarz and Proposition 6 we can write

[TABLE]

This inequality combined with (4.23) gives

[TABLE]

and thus the final result Theorem 5. ∎

5  Concentration of the matrix ${\cal L}$

The goal of this section is to prove Proposition 6. The proof is broken in two parts using the decomposition

[TABLE]

5.1 Thermal fluctuations

The first result, which is relation (4.10) in Proposition 6, expresses concentration w.r.t. the posterior distribution. It follows from concavity properties of the average free energy. Recall the notation $\mathbb{E}_{\lambda}[-]\equiv s_{n}^{-K(K+1)/2}\int_{{\cal D}_{n,K}}d\lambda\,[-]$.

Proof of (4.10) in Proposition 6.

By direct computation we have for any $(l,l^{\prime})\in\{1,\ldots,K\}^{2}$

[TABLE]

We have shown in (4.15) that

[TABLE]

We integrate the equality (5.2) over

[TABLE]

We obtain

[TABLE]

We have $|df_{n}/d\lambda_{ll^{\prime}}|\leq|\mathbb{E}[\langle x_{1l}\rangle\langle x_{1l^{\prime}}\rangle]|\leq S^{2}$ from (4.8), (4.9) so the first term is certainly smaller in absolute value than $2S^{2}/n$. Therefore

[TABLE]

In the last equality, the constants $C(S,K)$ are different. We now average this inequality w.r.t. to the remaining entries of $\lambda\in{\cal D}_{n,K}$, where the set ${\cal D}_{n,K}$ is defined by (3.2). This yields

[TABLE]

Summing all $K^{2}$ fluctuations for the various couples $(l,l^{\prime})$ yields the desired bound. ∎

5.2 Quenched fluctuations

The next proposition expresses concentration w.r.t. the quenched variables and is a consequence of the concentration of the free energy onto its average (w.r.t. the quenched variables). This is where the hypothesis (3.8) is crucial. This proposition together with (4.10) and relation (5.1) imply (4.11) in Proposition 6.

Proposition 7** (Quenched fluctuations of $\cal L$).**

Let $(s_{n})\in(0,1]^{\mathbb{N}}$ a positive sequence verifying $s_{n}\to 0_{+}$ and $s_{n}^{4}n\to+\infty$. Assume that the free energy concentrates as in identity (3.8). Then there exists $C(C_{f},K,S)>0$ s.t.

[TABLE]

Proof.

Let us define the non-averaged free energy:

[TABLE]

so that the averaged one given by (3.7) is simply $f_{n}(\lambda)=\mathbb{E}\,F_{n}(\lambda)$ with $\mathbb{E}=\mathbb{E}_{\theta}\mathbb{E}_{X|\theta_{0}}\mathbb{E}_{\widetilde{Y}|X,\theta_{\rm out}}\mathbb{E}_{Y|X,\lambda}$. We have the following identities: for any given realization of the quenched variables and any fixed $(l,l^{\prime})$,

[TABLE]

The same identities for $f_{n}(\lambda)=\mathbb{E}\,F_{n}(\lambda)$ are true but with an additional average $\mathbb{E}$ over the quenched variables (recall, e.g., (4.8)). The thermal fluctuations of $\cal L$ are almost directly equal to the the second derivative of $-\frac{1}{n}F_{n}(\lambda)$ as seen from (5.4), up to a lower order term that we consider now. We have

[TABLE]

where

[TABLE]

for some $C>0$. Indeed by (4.16) the entries of $\frac{d\lambda^{1/2}}{d\lambda_{ll^{\prime}}}$ are ${\cal O}(s_{n}^{-1/2})$, and thus $D_{n}=Cs_{n}^{-3/2}$ by the same manipulations as for getting (4.16). Therefore the following function of $\lambda_{ll^{\prime}}$

[TABLE]

is concave by construction, because thanks to (5.4), (5.5) we see that the second $\lambda_{ll^{\prime}}$-derivative of $F_{n,ll^{\prime}}(\lambda_{ll^{\prime}})$ is negative (for an appropriate positive constant $C$). Obviously its average

[TABLE]

is concave too. Concavity then allows to use the following standard lemma (see [28, 37] for a proof):

Lemma 8** (A bound for concave functions).**

Let $G(x)$ and $g(x)$ be concave functions. Let $\delta>0$ and define $C^{-}_{\delta}(x)\equiv g^{\prime}(x-\delta)-g^{\prime}(x)\geq 0$ and $C^{+}_{\delta}(x)\equiv g^{\prime}(x)-g^{\prime}(x+\delta)\geq 0$. Then

[TABLE]

First, from (5.6), (5.7) we have

[TABLE]

with

[TABLE]

Second, we obtain for the $\lambda_{ll^{\prime}}$-derivatives difference

[TABLE]

From (5.8) and (5.9) it is then easy to show that Lemma 8 applied to $G(x)\to F_{n,ll^{\prime}}(\lambda_{ll^{\prime}})$ and $g(x)\to f_{n,ll^{\prime}}(\lambda_{ll^{\prime}})$ with $x\to\lambda_{ll^{\prime}}$ gives

[TABLE]

where we used $|\lambda_{ll^{\prime}}|\leq(2K+1)s_{n}$ for any $(l,l^{\prime})\in\{1,\ldots,K\}^{2}$ as $\lambda\in{\cal D}_{n,K}$, and where

[TABLE]

Note that $\delta$ will be chosen small enough so that when $\lambda_{ll^{\prime}}$ is varied by $\pm\delta$ the matrix $\lambda$ remains in the set ${\cal D}_{n,K}$. Remark that by independence of the noise variables

[TABLE]

We square the identity (5.2) and take its expectation. Then using $(\sum_{i=1}^{p}v_{i})^{2}\leq p\sum_{i=1}^{p}v_{i}^{2}$ by convexity, and again that $|\lambda_{ll^{\prime}}|\leq(2K+1)s_{n}$ as well as the free energy concentration hypothesis (3.8) (irrelevant positive numerical constants are absorbed in the generic positive constants $C$),

[TABLE]

Recall $|C_{\delta}^{\pm}(\lambda_{ll^{\prime}})|=|f^{\prime}_{n,ll^{\prime}}(\lambda_{ll^{\prime}}\pm\delta)-f^{\prime}_{n,ll^{\prime}}(\lambda_{ll^{\prime}})|$. We have (here $Z\sim{\cal N}(0,1)$)

[TABLE]

using (4.8), (4.9) to assess that $|\mathbb{E}\langle{\cal L}_{ll^{\prime}}\rangle|\leq S^{2}$, and that $\lambda_{ll^{\prime}}$ is propositional to $s_{n}$ because $\lambda\in{\cal D}_{n,K}$. Thus we have the crude bound

[TABLE]

Consider again the integration domain (5.3). We reach

[TABLE]

The mean value theorem and (5.12) imply

[TABLE]

Therefore

[TABLE]

Averaging (5.11) over $\lambda_{ll^{\prime}}\in(a_{n},b_{n})$ and choosing $\delta=\delta_{n}$ s.t. $\delta_{n}/s_{n}\to 0_{+}$ yields

[TABLE]

We optimize the bound by choosing $\delta_{n}=s_{n}^{2/3}n^{-1/3}$ (which indeed verifies $\delta_{n}/s_{n}\to 0$ as long as $s_{n}n\to+\infty$). The desired result is then obtaind after averaging over the remaining $K(K+1)/2-1$ independent entries of $\lambda\in{\cal D}_{n,K}$, and summing the $K^{2}$ fluctuations for the various $(l,l^{\prime})$ couples. ∎

Acknowledgments

Funding from the “Chaire de recherche sur les modèles et sciences des données”, Fondation CFM pour la Recherche-ENS is acknowledged. I would like to deeply thank Nicolas Macris, Antoine Maillard and Florent Krzakala for stimulating exchanges. In particular Nicolas Macris for the very many discussions leading to the idea of the perturbation described in section 3, his careful reading of the manuscript, but most of all for his constant support. I am also very grateful to Léo Miolane for pointing identity (4.18), and to Clément Luneau for his careful reading and corrections of the manuscript. Last but not least, I want to express all my gratitude to Dmitry Panchenko for helping me facing conceptual issues on related questions, and his detailed and patient explanations on some of his works.