Concentration of the matrix-valued minimum mean-square error in optimal Bayesian inference

TL;DR

This paper proves that the matrix-valued minimum mean-square error in Bayesian inference concentrates as the problem size grows, extending techniques from spin glass physics to various inference models.

Contribution

It introduces a concentration result for the MMSE in vector-valued Bayesian inference, applicable to models like spiked matrices, tensors, and neural networks, under known model parameters.

Findings

MMSE concentrates in large-scale Bayesian inference models

Applicable to spiked matrix and tensor models, neural networks, and generalized linear models

Provides theoretical foundation for mutual information formulas in these settings

Abstract

We consider Bayesian inference of signals with vector-valued entries. Extending concentration techniques from the mathematical physics of spin glasses, we show that the matrix-valued minimum mean-square error concentrates when the size of the problem increases. Such results are often crucial for proving single-letter formulas for the mutual information when they exist. Our proof is valid in the optimal Bayesian inference setting, meaning that it relies on the assumption that the model and all its hyper-parameters are known. Examples of inference and learning problems covered by our results are spiked matrix and tensor models, the committee machine neural network with few hidden neurons in the teacher-student scenario, or multi-layers generalized linear models.

Equations

$$P(X=x|\tilde{Y},\theta )=\frac{P_0(x|{\theta }_0)P_{\mathrm{out}}(\tilde{Y}|x,{\theta }_{\mathrm{out}})}{\int d{P}_0(x|{\theta }_0)P_{\mathrm{out}}(\tilde{Y}|x,{\theta }_{\mathrm{out}})}.$$

$$\mathbb{E}{F}_{0,n}\equiv \frac{1}{n}H(\tilde{Y}|\theta )=-\frac{1}{n}\mathbb{E}\ln \int d{P}_0(x|{\theta }_0)P_{\mathrm{out}}(\tilde{Y}|x,{\theta }_{\mathrm{out}}).$$

$$Q_{k{k}^{\prime }}\equiv \frac{1}{n}(X^{⊺}x{)}_{k{k}^{\prime }}=\frac{1}{n}{\sum }_{i=1}^n{X}_{ik}{x}_{i{k}^{\prime }}.$$

$$\textstyle\frac{1}{n}\mathbb{E}\big{[}(X-\langle x\rangle_{0})^{\intercal}(X-\langle x\rangle_{0})\big{]}=\mathbb{E}[X_{1}X_{1}^{\intercal}]-\mathbb{E}\langle Q\rangle_{0}$$

$$\frac{1}{n}\mathbb{E}\Vert X-⟨x{⟩}_0{\Vert }_{\mathrm{F}}^2=\mathbb{E}\Vert X_1{\Vert }^2-\mathrm{Tr}\mathbb{E}⟨Q{⟩}_0$$

$$\textstyle\frac{1}{n^{2}}\mathbb{E}\|XX^{\intercal}\!-\!\langle xx^{\intercal}\rangle_{0}\|^{2}_{\rm F}=\mathbb{E}\big{[}(X_{1}^{\intercal}X_{2})^{2}\big{]}\!-\!\mathbb{E}\big{\langle}\|Q\|^{2}_{\rm F}\big{\rangle}_{0}\!+\!{\cal O}(\frac{1}{n}).$$

$${\tilde{Y}}_{i_1\dots i_p}=n^{\frac{1-p}{2}}{\sum }_{k=1}^K{X}_{i_{1}k}{X}_{i_{2}k}\dots X_{i_{p}k}+{\tilde{Z}}_{i_1\dots i_p}$$

$$\textstyle\widetilde{Y}_{\mu}\sim p_{\rm out}\big{(}\cdot\big{|}\sum_{i=1}^{n}\theta_{\mu i}X_{i}\big{)},\qquad 1\leq\mu\leq m=\Theta(n).$$

$${\tilde{Y}}_{\mu }=\text{ sign}{\sum }_{k=1}^K\text{ sign}{\sum }_{i=1}^n{\theta }_{\mu i}{X}_{ik},1\le \mu \le m.$$

$$\displaystyle\begin{cases}X^{(L)}_{i_{L}}\sim p_{\rm out}^{(L)}\big{(}\cdot\big{|}\sum_{j=1}^{n_{L-1}}\theta_{i_{L}j}^{(L)}X_{j}^{(L-1)}\big{)},\\ X_{i_{L-1}}^{(L-1)}\sim p_{\rm out}^{(L-1)}\big{(}\cdot\big{|}\sum_{j=1}^{n_{L-2}}\theta_{i_{L-1}j}^{(L-1)}X_{j}^{(L-2)}\big{)},\\ \qquad\qquad\qquad\qquad\vdots\\ X_{i_{1}}^{(1)}\sim p_{\rm out}^{(1)}\big{(}\cdot\big{|}\sum_{j=1}^{n_{0}}\theta_{i_{1}j}^{(1)}X_{j}^{(0)}\big{)},\end{cases}$$

$$\displaystyle\begin{cases}\widetilde{Y}_{ij}=n^{-1/2}\,\sum_{k=1}^{K}X_{ik}X_{jk}+Z_{ij}\,,&1\leq i\leq j\leq n\,,\\ X_{i}\sim p_{\rm out}\big{(}\cdot\big{|}\sum_{j=1}^{n_{0}}\theta_{ij}X_{j}^{(0)}\big{)}\,,&1\leq i\leq n\,.\end{cases}$$

$$Y=X{\lambda }_n^{1/2}+Z,\text{i.e.,}{Y}_i={\lambda }_n^{1/2}{X}_i+Z_i,1\le i\le n.$$

$${\mathcal{D}}_K\equiv$$

$$\textstyle\qquad\qquad\qquad\qquad\qquad\tilde{\lambda}_{kk}\in(2K,2K+1)\ \forall\ k\big{\}}$$

$$\{\begin{array}{ll}\tilde{Y}& \sim P_{\mathrm{out}}(\cdot |X,{\theta }_{\mathrm{out}}),\\ Y& =X{\lambda }_n^{1/2}+Z.\end{array}$$

$$\frac{P_0(x|{\theta }_0)P_{\mathrm{out}}(\tilde{Y}|x,{\theta }_{\mathrm{out}})e^{-{\mathcal{H}}_{{\lambda }_n}(x,Y)}}{\int d{P}_0(x|{\theta }_0)P_{\mathrm{out}}(\tilde{Y}|x,{\theta }_{\mathrm{out}})e^{-{\mathcal{H}}_{{\lambda }_n}(x,Y)}}.$$

$$F_n({\lambda }_n)\equiv -\frac{1}{n}\ln \int d{P}_0(x|{\theta }_0)P_{\mathrm{out}}(\tilde{Y}|x,{\theta }_{\mathrm{out}})e^{-{\mathcal{H}}_{{\lambda }_n}(x,Y)}.$$

$$\textstyle\mathbb{E}\big{[}\big{(}F_{n}(\lambda_{n})-\mathbb{E}\,F_{n}(\lambda_{n})\big{)}^{2}\big{]}\leq C_{F}\,n^{-1}\,.$$

$$\textstyle\mathbb{E}_{\lambda}\mathbb{E}\big{\langle}\|Q-\langle Q\rangle\|_{\rm F}^{2}\big{\rangle}$$

$$\textstyle\mathbb{E}_{\lambda}\mathbb{E}\big{\langle}\|Q-\mathbb{E}\langle Q\rangle\|_{\rm F}^{2}\big{\rangle}\leq C(s_{n}^{4}n)^{-1/6}\,.$$

$$\textstyle\mathbb{E}_{\lambda}\mathbb{E}\big{\langle}\|{\cal L}-\langle{\cal L}\rangle\|_{\rm F}^{2}\big{\rangle}$$

$$\textstyle\mathbb{E}_{\lambda}\mathbb{E}\big{\langle}\|{\cal L}-\mathbb{E}\langle{\cal L}\rangle\|_{\rm F}^{2}\big{\rangle}\leq C(s_{n}^{4}n)^{-1/3}\,.$$

$$\textstyle\mathbb{E}_{\lambda}\mathbb{E}\big{\langle}(Q_{ll^{\prime}}-\langle Q_{ll^{\prime}}\rangle)^{2}\big{\rangle}=\mathbb{E}_{\lambda}\mathbb{E}\langle Q_{ll^{\prime}}^{2}\rangle-\mathbb{E}_{\lambda}\mathbb{E}\big{[}\langle Q_{ll^{\prime}}\rangle^{2}\big{]}$$

$$\textstyle=\frac{1}{n^{2}}\sum_{i,j=1}^{n}\mathbb{E}_{\lambda}\mathbb{E}\big{[}X_{il}X_{jl}(\langle x_{il^{\prime}}x_{jl^{\prime}}\rangle-\langle x_{il^{\prime}}\rangle\langle x_{jl^{\prime}}\rangle)\big{]}$$

$$\textstyle\leq C\big{\{}\frac{1}{n^{2}}\sum_{i,j=1}^{n}\mathbb{E}_{\lambda}\mathbb{E}\big{[}(\langle x_{il^{\prime}}x_{jl^{\prime}}\rangle-\langle x_{il^{\prime}}\rangle\langle x_{jl^{\prime}}\rangle)^{2}\big{]}\big{\}}^{1/2}$$

$$\textstyle\frac{1}{n^{2}}\sum_{i,j=1}^{n}\mathbb{E}\big{[}(\langle x_{il}x_{jl}\rangle-\langle x_{il}\rangle\langle x_{jl}\rangle)^{2}\big{]}$$

$$+C(s_{n}n{)}^{-1}.$$

$$\textstyle\mathbb{E}_{\lambda}{\rm Tr}\,\mathbb{E}\big{\langle}Q({\cal L}-\mathbb{E}\langle{\cal L}\rangle)\big{\rangle}\!=\!-\sum_{l\neq l^{\prime}}\mathbb{E}_{\lambda}\mathbb{E}\big{\langle}(Q_{ll^{\prime}}-\mathbb{E}\langle Q_{ll^{\prime}}\rangle)^{2}\big{\rangle}$$

$$\textstyle-\frac{1}{2}\sum_{l}\mathbb{E}_{\lambda}\mathbb{E}\big{\langle}(Q_{ll}-\mathbb{E}\langle Q_{ll}\rangle)^{2}\big{\rangle}\!+\!\mathbb{E}_{\lambda}{\rm Tr}\,\mathbb{E}\big{\langle}Q(\langle Q^{(12)}\rangle-Q)\big{\rangle}$$

$$\textstyle+\frac{1}{2}\sum_{l}\mathbb{E}_{\lambda}\mathbb{E}\big{\langle}Q_{ll}(Q_{ll}-\langle Q_{ll}^{(12)}\rangle)\big{\rangle}\pm C(s_{n}n)^{-1/4}.$$

$$\textstyle\big{|}\mathbb{E}_{\lambda}{\rm Tr}\,\mathbb{E}\big{\langle}Q({\cal L}-\mathbb{E}\langle{\cal L}\rangle)\big{\rangle}\big{|}$$

$$\le C(s_n^{4}n{)}^{-1/6}.$$

$$\textstyle\mathbb{E}_{\lambda}\mathbb{E}\big{\langle}\|Q-\langle Q^{(12)}\rangle\|_{\rm F}^{2}\big{\rangle}$$

Full text

I INTRODUCTION

This decade is witnessing a burst of mathematical studies related to inference and learning problems. One reason is that an important arsenal of methods, developed in particular in the context of spin glass physics, has found a new rich playground where it can be applied with success [1, 2, 3, 4]. In particular important progress has been made recently in the context of high-dimensional Bayesian inference and learning. Examples of problems in this class include spiked matrix and tensor models [5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18], random linear and generalized estimation [19, 20, 21, 22, 23, 24], models of neural networks in the teacher-student scenario [23, 25, 26], or sparse graphical models such as error-correcting codes and block models [27, 28, 29].

All these results are based in some way or another on the control of the fluctuations of the overlap, which is related up to a constant to the minimum mean-square error. Optimal Bayesian inference –optimal meaning that the true posterior is known– is an ubiquitous setting in the sense that the overlap can be shown to concentrate, and this in the whole regime of parameters (amplitude of the noise, number of observations/data points divided by the number of parameters to infer etc). When the overlap concentrates (and the problem is “random enough”) one expects single-letter variational formulas for the asymptotic mutual information.

In many statistical models the overlap is a scalar. In the context of optimal Bayesian inference it is now quite standard to show that the scalar overlap is self-averaging, see, e.g., [30, 14]. The techniques to do so are coming from communications starting with [31, 32, 19] (and then generalized in [33, 27]), and are extensions of methods used in the analysis of spin glasses [1, 34, 35, 3, 36]. In this paper we consider instead Bayesian inference of signals made of vectorial components in which case the overlap is a matrix. The concentration techniques developed for scalar overlaps do not apply directly, and need to be extended using new non-trivial ideas. Examples of inference problems where matrix overlaps appear are the factorization of matrices and tensors of rank greater than one [11], or the committee machine neural network [37, 38, 39, 25]. They also appear in the context of spin glasses [40, 41, 42].

II OPTIMAL INFERENCE OF TALL MATRICES

II-A General setting

All quantities in this paper are real. Consider a model where a “tall” matrix-signal $X=(X_{ik})\in[-S,S]^{n\times K}$ made of $n$ components, that are each a $K$-dimensional vector where $K\ll n$ is independent of $n$, is generated probabilistically. Its probability prior distribution $P_{0}$ may depend on a generic hyper-parameter $\theta_{0}\in\Theta_{0}$, i.e., $X\sim P_{0}(\,\cdot\,|\theta_{0})$. Data (also called observations) $\widetilde{Y}$ are then generated conditionally on the unknown $X$ and an hyper-parameter $\theta_{\rm out}\in\Theta_{\rm out}$. Namely, the data $\widetilde{\cal Y}\ni\widetilde{Y}\sim P_{\rm out}(\,\cdot\,|X,\theta_{\rm out})$, with $\widetilde{\cal Y}$ a generic set: the data and hyper-parameters can be vectors, tensors etc, so that our model is very general. We assume that $\theta_{0}$ and $\theta_{\rm out}$ are also random, with distributions $P_{\theta_{0}},P_{\theta_{\rm out}}$.

The task is to infer the signal $X$ given the data $\widetilde{Y}$. We moreover assume that the hyper-parameters $\theta\equiv(\theta_{0},\theta_{\rm out})$, the kernel $P_{\rm out}$ and the prior $P_{0}$ are known to the statistician, and call this setting optimal Bayesian inference.

The information-theoretical optimal way of infering the signal follows from its posterior distribution. Using Bayes’ formula the posterior for the base inference model reads

[TABLE]

The averaged free energy (i.e., the Shannon entropy density of the data given the hyper-parameters) equals

[TABLE]

The average $\mathbb{E}=\mathbb{E}_{\theta}\mathbb{E}_{X|\theta_{0}}\mathbb{E}_{\widetilde{Y}|X,\theta_{\rm out}}$ is over $(\theta,X,\widetilde{Y})$, jointly called the quenched variables as they are fixed by the realization of the problem, in contrast with the dynamical variable $x$ which fluctuates according to the posterior. We call model (1) the “base model” in contrast with the perturbed model presented in section III, a slightly modified version of the base model where additional side-information is given, and for which concentration results can be proved without altering the $n\to+\infty$ limit of the averaged free energy, see Lemma 1.

The central object of interest is the $K\times K$ overlap matrix (or simply overlap) $Q=(Q_{kk^{\prime}})$ defined as

[TABLE]

Here $x$ is a sample from the posterior and $X$ is the signal. The overlap contains a lot of information. Using that the estimator minimizing the mean-square error is the posterior mean $\langle x\rangle_{0}\equiv\mathbb{E}[X|\widetilde{Y},\theta]$ (denoting $\langle-\rangle_{0}$ the expectation w.r.t. the posterior (1) of the base model), the $K\times K$ matrix-valued minimum mean-square error (MMSE) is

[TABLE]

where $X_{i}^{\intercal}\in\mathbb{R}^{K}$ is a row of $X$ (all vectors are columns, including rows of matrices considered alone, transposed vectors are rows). The scalar MMSE is then simply

[TABLE]

($\rm Tr$ is the trace). Another metric for problems where, e.g., the sign of the signal is lost due to symmetries is

[TABLE]

II-B Examples

In the symmetric order-$p$ rank-$K$ tensor factorization problem, the data-tensor $\widetilde{Y}=(\widetilde{Y}_{i_{1}\ldots i_{p}})$ is generated as

[TABLE]

for $1\leq i_{1}\leq i_{2}\leq\ldots\leq i_{p}\leq n$. Here $\widetilde{Z}$ is a Gaussian noise tensor with independent and identically distributed (i.i.d.) ${\cal N}(0,1)$ entries, and the signal components are i.i.d., i.e., with a prior $P_{0}=p_{0}^{\otimes n}$ with $p_{0}$ supported on $[-S,S]^{K}$. The case $p=2$ is the spiked Wigner model and is one of the simplest probabilistic model for principal component analysis [5]. In both the analysis of [17, 18] for this problem the matrix overlap concentration is a key result.

Another model is the following generalized linear model (GLM) (recall $X_{i}\in\mathbb{R}^{K}$):

[TABLE]

Given $\mathbb{R}^{m\times n}\ni\theta_{\rm out}=(\theta_{\mu})_{\mu=1}^{m}$ and $X$, the $m$ data points are i.i.d., thus the notation $p_{\rm out}$ instead of $P_{\rm out}(\,\cdot\,|\theta_{\rm out}X)=\otimes_{\mu=1}^{m}p_{\rm out}(\,\cdot\,|X^{\intercal}\theta_{\mu})$. We also assume that the prior $P_{0}=p_{0}^{\otimes n}$. A particular simple deterministic case is

[TABLE]

This model is a version of the committee machine [23, 25]. Here $(X_{ik})_{i=1}^{n}$ can be interpreted as the weights of the $k$-th hidden neuron, and $(\theta_{\mu})$ are $n$-dimensional data points used to generate the labels $(\widetilde{Y}_{\mu})$. The teacher-student scenario in which our results apply is: the teacher network (4) (or (3) in general) generates $\widetilde{Y}$ from the data $\theta_{\rm out}$. The pairs $(\widetilde{Y}_{\mu},\theta_{\mu})$ are then used in order to train (i.e., learn the weights of) a student network with the same architecture.

A richer example is a multi-layer version of the GLM:

[TABLE]

where each index $i_{\ell}$ runs from $1$ to $n_{\ell}=\Theta(n_{0})$, for $\ell=1,\ldots,L$. The input $X^{(0)}\sim P_{0}$ is factorized as $P_{0}=p_{0}^{\otimes n_{0}}$. In this model $(X^{(\ell)})_{\ell=1}^{L-1}$ represent intermediate hidden variables, the visible variable $X^{(L)}=\widetilde{Y}$ is the data, and $\theta_{\rm out}=(\theta^{(\ell)})$ with $\theta^{(\ell)}$ representing the weight matrix at the $\ell$-th layer. Note that in the single layer case (3), $\theta_{\rm out}$ was interpreted as data points and $X$ as the weight vector to learn. This model has been studied by various authors when $K=1$ and when the components $(X_{j}^{(\ell)})$ are scalars [43, 44, 45, 26, 46]. But one can define generalizations where these are multi-dimensional, in which case overlap matrices naturally appear.

A last example could be another combinaison of statistical models such as a spiked Wigner model where the hidden low-rank representation $X$ of the data $\widetilde{Y}$ has a complex generative prior. For example $X$ could be generated from a GLM over a more primitive signal $X^{(0)}\sim P_{0}=p_{0}^{\otimes n_{0}}$ (here $n=\Theta(n_{0})$):

[TABLE]

This set-up has recently attracted attention [47] for studying models of complex structured data with generative priors.

III THE PERTURBED MODEL, AND RESULTS

III-A The vectorial Gaussian channel perturbation

In order to “force” overlap concentration we need, in addition to the data $\widetilde{Y}$, infinitesimal side-information $Y\in\mathbb{R}^{n\times K}$ about $X$ coming from a vectorial Gaussian channel:

[TABLE]

The i.i.d. Gaussian noise $(Z_{i})\sim{\cal N}(0,I_{K})^{\otimes n}$. The signal-to-noise (SNR) matrix controlling the signal strength $\lambda_{n}\equiv s_{n}\tilde{\lambda}$ with a sequence $(s_{n})$ that tends to $0_{+}$, and $\tilde{\lambda}$ belongs to

[TABLE]

(other sets could be used but this one is convenient for the proof). We also denote ${\cal D}_{n,K}\equiv s_{n}{\cal D}_{K}$ so that $\lambda_{n}\in{\cal D}_{n,K}$. Matrices belonging to ${\cal D}_{n,K}$ are symmetric strictly diagonally dominant with positive entries and thus ${\cal D}_{n,K}\subset{\cal S}_{K}^{+}$, where ${\cal S}_{K}^{+}$ is the set of symmetric positive definite matrices of dimension $K\times K$, see [48]. As $\lambda_{n}\in{\cal D}_{n,K}$ it has a unique square root matrix that we denote $\lambda_{n}^{1/2}=\sqrt{s_{n}}\,\tilde{\lambda}^{1/2}$.

The perturbed inference model is then

[TABLE]

It is called “perturbed model” because the base model has been slightly modified by adding new data points coming from (6) that are “weak” (as $s_{n}\to 0_{+}$). The posterior $P(X=x|\widetilde{Y},Y,\theta,\lambda_{n})$ of the perturbed model reads

[TABLE]

where ${\cal H}_{\lambda_{n}}(x,Y)\equiv\frac{1}{2}\|x\lambda_{n}^{1/2}\|_{\rm F}^{2}-{\rm Tr}(Y^{\intercal}x\lambda_{n}^{1/2})$. We define the bracket $\langle-\rangle$ as the expectation w.r.t. the posterior of the perturbed model: $\langle g\rangle\equiv\int dP(X=x|\widetilde{Y},Y,\theta,\lambda_{n})\,g(x)$. Thus $\langle g\rangle$ depends on $(\widetilde{Y},Y,\theta)$ and the SNR $\lambda_{n}$.

It is crucial to notice that the perturbed model (7) is set in the optimal Bayesian inference setting. Again, this means that in addition to the data $(\widetilde{Y},Y)$ the statistician knows the data generating model, namely the kernel $P_{\rm out}$ and the additive Gaussian nature of the noise in the second channel in (7), the prior $P_{0}$ as well as all hyper-parameters $(\theta,\lambda_{n})$, and is therefore able to write the true posterior (8).

An important object is the free energy of model (7):

[TABLE]

Concentration of the overlap requires an hypothesis:

Hypothesis 1** (Free energy concentration)**

There exists a constant $C_{F}$ that may depend on everything but $n$, and s.t.

[TABLE]

The expectation $\mathbb{E}\equiv\mathbb{E}_{\theta}\mathbb{E}_{X|\theta_{0}}\mathbb{E}_{\widetilde{Y}|X,\theta_{\rm out}}\mathbb{E}_{Y|X,\lambda_{n}}$ is over all quenched variables but not over $\lambda_{n}$, which remains fixed.

For purely generic optimal inference models without any restrictions on the distributions $(P_{0},P_{\rm out},P_{\theta_{0}},P_{\theta_{\rm out}})$ it is generally very hard, if not wrong, to try proving (9). The model must be “random enough” and possess some underlying factorization structure for such hypothesis to be true (thus the factorization properties assumed in the examples of section II-B). The most studied case in the literature is when the prior and the kernel factorize, namely $P_{0}=p_{0}^{\otimes n}$ and the data points are i.i.d. given $(X,\theta_{\rm out})$. The examples (2)–(4) fall in this class. Under such factorization assumptions it is quite straightforward to prove (9) using standard techniques (see, e.g., [14, 23]). But such simple factorization properties are not always there, as illustrated by the two last examples in section II-B. In these examples it is a perfectly valid question to wonder wether the overlap of the hidden variables do concentrate111Note that proving concentration of the overlap for a hidden variable requires a perturbation of the form (6) over the hidden variable, not over $X^{(0)}$, which in this case is just interpreted as a constitutive element of the prior of the hidden variable of interest, see [26] where this is done. (this question is crucial in the analysis of [26]). The hidden variables have very complex structured prior (i.e., probability distribution), with highly non-trivial factorization properties, in which case proving (9) requires work. See, e.g., [26] where this has been done for the multi-layer GLM (5) with a single hidden layer ($L=2$), where this is already challenging.

An important feature of the perturbation is that it does not change the limit of the averaged free energy; this means that in a certain sense the perturbed model is equivalent to the base one at a “macroscopic” level, i.e., for the global quantities. We denote in this paper $C$ a generic constant that may depend on all parameters in the problem like $K$ and $S$ but not on $n$.

Lemma 1** (Free energy equivalence)**

There exists a constant $C$ s.t. $|\mathbb{E}\,F_{0,n}-\mathbb{E}\,F_{n}(\lambda_{n})|\leq Cs_{n}$. Thus $\mathbb{E}\,F_{0,n}$ and $\mathbb{E}\,F_{n}(\lambda_{n})$ have same $n\to+\infty$ limit, provided it exists.

III-B Main results

Our main results are concentration theorems for the overlap in a (perturbed) model of optimal Bayesian inference. We start with the first type of fluctuations, namely the fluctuations of the overlap w.r.t. the posterior distribution, or what is called “thermal fluctuations” in statistical mechanics. Controlling these fluctuations does not require that the free energy concentrates (the hypothesis (9) is not required). Denote $\mathbb{E}_{\lambda}\,g\equiv{\rm Vol}({\cal D}_{n,K})^{-1}\int_{{\cal D}_{n,K}}d\lambda_{n}\,g(\lambda_{n})$ the average over the perturbation matrix $\lambda_{n}$, where ${\rm Vol}({\cal D}_{n,K})=s_{n}^{K(K+1)/2}$ ($\lambda_{n}$ has $K(K+1)/2$ independent entries). Then:

Theorem 2** (Thermal fluctuations of $Q$)**

Consider an optimal Bayesian inference problem (i.e., for which the true posterior is known), with side information coming from the channel (6); i.e., a model of the form (7). Let $(s_{n})$ a sequence s.t. $s_{n}\to 0_{+}$ and $s_{n}n\to+\infty$. There exists $C>0$ s.t.

[TABLE]

The next, stronger, result takes care of the additional fluctuations due to the quenched randomness, and requires this time the free energy concentration hypothesis:

Theorem 3** (Total fluctuations of $Q$)**

Consider a perturbed optimal Bayesian inference problem of the form (7). Assume Hypothesis 1. Let $(s_{n})$ verify $s_{n}\to 0_{+}$ and $s_{n}^{4}n\to+\infty$. There exists $C=C(C_{f},K,S)>0$ s.t.

[TABLE]

We emphasize that, as Theorem 2 does not require Hypothesis 1, it is valid very generically, even for very complex models without any factorization properties for the signal’s prior nor for the kernel; it is only a consequence of the perturbation and the Bayesian optimality. In such models, deriving single-letter formulas for quantities like the mutual information or the MMSE is doomed (as generally there is not). Indeed, proofs of such simple formulas always require in one way or another strong factorization properties, directly, like, e.g., in [11, 12], or indirectly through the need of the stronger concentration result Theorem 3 as in [14, 13, 23, 17, 16, 18]222In these papers the concentration needs to be proven for an appropriate “interpolating” model..

Another remark is related to the role of the perturbation (i.e., side-information). Our theorems require an external average $\mathbb{E}_{\lambda}$ over the perturbation: this is not an artefact of the proof. Indeed, there might be a (zero-measure) set in the hyper-parameters space $\Theta_{0}\times\Theta_{\rm out}$ where, in the $n\to+\infty$ limit, there are phase transitions. Phase transitions manifest themselves in particular by a non self-averaging behavior of the overlap. But averaging over a vanishing window of $\lambda_{n}$, which importantly is independent of $\theta$, allows to “smoothen” the overlap fluctuations, effectively cancelling the dramatic effect of possible phase transitions.

IV PROOF IDEA

We give few pointers to help the reader to get idea of the proof. All details can be found in [49].

Proving overlap concentration relies on the concentration of another $K\times K$ matrix ${\cal L}\equiv\frac{1}{n}\nabla_{\lambda_{n}}{\cal H}_{\lambda_{n}}(x,Y)$.

Proposition 4** (Concentration of ${\cal L}$)**

Let $s_{n}\to 0_{+}$ and $s_{n}n\to+\infty$. Then there is $C>0$ s.t.

[TABLE]

If $s_{n}^{4}n\to+\infty$ and Hypothesis 1 is verified,

[TABLE]

The fluctuations of this matrix are easier to control than the ones of the overlap because ${\cal L}$ is related to the $\lambda_{n}$-gradient of the free energy, which is self-averaging by hypothesis (9). The proof is a straightforward extension to the matrix case of the one found in [30, 23] and requires no new ideas. This general result does not depend on the fact that we consider optimal Bayesian inference; it is only a consequence of the perturbation, i.e., the side information coming from the channel (6). What instead does require new ideas and relies on the Bayesian optimal setting is the link between the concentration of ${\cal L}$ and the one of $Q$. The additional difficulty w.r.t. what is done in [30, 23] for a scalar overlap (i.e., the case $K=1$) is that the matrix $Q$ is not symmetric, even if its expectation $\mathbb{E}\langle Q\rangle$ is. Symmetry in expectation is a consequence of the general identity (sometimes called “Nishimori identity”) \mathbb{E}\big{\langle}g(x,X;\widetilde{Y},Y)\big{\rangle}=\mathbb{E}\big{\langle}g(x,x^{\prime};\widetilde{Y},Y)\big{\rangle}, where $X$ is the signal, $x,x^{\prime}$ are i.i.d. samples from the posterior (8), $\langle-\rangle$ is the expectation w.r.t. the product posterior measure, and $g$ is any bounded function. This innocent-looking key identity on which relies the whole proof follows directly from Bayes’ law –thus the importance of placing ourselves in the Bayesian optimal setting–, see [30, 23]. Applied to $\mathbb{E}\langle Q\rangle=\mathbb{E}[X^{\intercal}\langle x\rangle]=\mathbb{E}\langle(x^{\prime})^{\intercal}x\rangle=\mathbb{E}[\langle x\rangle^{\intercal}\langle x\rangle]$ which is indeed symmetric.

Linking the thermal fluctuations of $\cal L$ and $Q$: Let us start by giving the main steps behind the proof of Theorem 2. The key insight is the following inequality: by definition of the overlap (and for any $l,l^{\prime}\in\{1,\ldots,K\}$),

[TABLE]

for some $C>0$ using Cauchy-Schwarz, and that the prior has bounded support. Combining the Nishimori identity, Gaussian integration by parts and by careful algebra using the formula $\frac{d\lambda}{d\lambda_{ll^{\prime}}}=\lambda^{1/2}\frac{d\lambda^{1/2}}{d\lambda_{ll^{\prime}}}+\frac{d\lambda^{1/2}}{d\lambda_{ll^{\prime}}}\lambda^{1/2}$ one can show

[TABLE]

Identity (10) in Proposition (4) for $\cal L$ then implies that the $x_{il}$’s asymptotically “decouple”. When this is plugged in (12), this decoupling property translates into Theorem 2.

Total fluctuations of $Q$: We now consider Theorem 3, which requires to obtain Theorem 2 first. The proof ressembles the derivation of the Ghirlanda-Guerra identities in the context of spin glasses [1]. The main identity is

[TABLE]

Here $Q^{(12)}\equiv x^{\intercal}x^{\prime}$ is the overlap between two i.i.d. samples from the (perturbed) posterior (8), so that $\langle Q^{(12)}\rangle=\langle x\rangle^{\intercal}\langle x\rangle$ is symmetric. The relation (13) is shown by the Nishimori identity and Gaussian integration by parts, which in particular allows to prove $\mathbb{E}\langle{\cal L}\rangle=\frac{1}{2}{\rm diag}(\mathbb{E}\langle Q\rangle)-\mathbb{E}\langle Q\rangle$, as well as the use of Theorem 2. Again, in the derivation one has to be careful as the matrices that appear are not symmetric, which complicates the task.

Now, by (11) in Proposition (4) and the Cauchy-Schwarz inequality we have that the left hand side of (13) verifies

[TABLE]

Therefore, because of the alternating signs on the right hand side of (13), showing that \mathbb{E}_{\lambda}\mathbb{E}\big{\langle}\|Q-\mathbb{E}\langle Q\rangle\|_{\rm F}^{2}\big{\rangle} is small requires to prove that the third and forth terms are small. These can be thought of as a “measure of asymmetry” of the overlap matrix $Q$. The last crucial step is therefore showing (this again relies on the Nishimori identity)

[TABLE]

ACKNOWLEDGMENTS

Funding from Fondation CFM pour la Recherche-ENS is acknowledged. I would like to thank Nicolas Macris, Dmitry Panchenko, Antoine Maillard, Florent Krzakala, Léo Miolane and Clément Luneau for discussions.