Mutual Information and Optimality of Approximate Message-Passing in Random Linear Estimation

TL;DR

This paper rigorously establishes the mutual information and optimality of approximate message-passing algorithms in Gaussian linear estimation, showing they achieve minimal mean-square error in relevant scenarios.

Contribution

It provides a rigorous proof that the replica formula bounds the mutual information and demonstrates the optimality of AMP algorithms outside the hard phase.

Findings

Replica formula bounds mutual information from above.

AMP algorithms reach minimal MSE outside the hard phase.

No algorithmically hard phase exists in these systems.

Abstract

We consider the estimation of a signal from the knowledge of its noisy linear random Gaussian projections. A few examples where this problem is relevant are compressed sensing, sparse superposition codes, and code division multiple access. There has been a number of works considering the mutual information for this problem using the replica method from statistical physics. Here we put these considerations on a firm rigorous basis. First, we show, using a Guerra-Toninelli type interpolation, that the replica formula yields an upper bound to the exact mutual information. Secondly, for many relevant practical cases, we present a converse lower bound via a method that uses spatial coupling, state evolution analysis and the I-MMSE theorem. This yields a single letter formula for the mutual information and the minimal-mean-square error for random Gaussian linear estimation of all discrete bounded signals. In addition, we prove that the low complexity approximate message-passing algorithm is optimal outside of the so-called hard phase, in the sense that it asymptotically reaches the minimal-mean-square error. In this work spatial coupling is used primarily as a proof technique. However our results also prove two important features of spatially coupled noisy linear random Gaussian estimation. First there is no algorithmically hard phase. This means that for such systems approximate message-passing always reaches the minimal-mean-square error. Secondly, in a proper limit the mutual information associated to such systems is the same as the one of uncoupled linear random Gaussian estimation.

Figures

Equations

$$y_{\mu }=\sum _{i=1}^N{\varphi }_{\mu i}{s}_i+z_{\mu }\sqrt{\Delta }.$$

$$\displaystyle P^{\rm cs}({\textbf{s}},{\textbf{y}}|\bm{\phi})=(2\pi\Delta)^{-M/2}\exp\Big{(}-\frac{1}{2\Delta}\sum\limits_{\mu=1}^{M}([\bm{\phi}{\textbf{s}}]_{\mu}-y_{\mu})^{2}\Big{)}\prod_{l=1}^{L}P_{0}({\textbf{s}}_{l})\,.$$

$$P^{\mathrm{cs}}(\text{s}=\text{x}|\text{y},\mathbf{\varphi })=P^{\mathrm{cs}}(\text{x}|\text{y},\mathbf{\varphi })$$

$$\displaystyle\mathcal{Z}^{\rm cs}({\textbf{y}},\bm{\phi})=\int d{\textbf{x}}\exp\Big{(}-\frac{1}{2\Delta}\sum\limits_{\mu=1}^{M}([\bm{\phi}{\textbf{x}}]_{\mu}-y_{\mu})^{2}\Big{)}\prod_{l=1}^{L}P_{0}({\textbf{x}}_{l})$$

$$P^{\mathrm{cs}}(\text{y}|\mathbf{\varphi })={\mathcal{Z}}^{\mathrm{cs}}(\text{y},\mathbf{\varphi })(2\pi \Delta {)}^{-M/2}.$$

$$i^{\mathrm{cs}}$$

$$=-\frac{1}{L}h(\text{Y}|\text{S},\mathbf{\Phi })+\frac{M}{2L}\ln (2\pi \Delta )-\frac{1}{L}{\mathbb{E}}_{\mathbf{\Phi },\text{Y}}[\ln ({\mathcal{Z}}^{\mathrm{cs}}(\text{Y},\mathbf{\Phi }))]$$

$$=-\frac{\alpha B}{2}-\frac{1}{L}{\mathbb{E}}_{\mathbf{\Phi },\text{Y}}[\ln ({\mathcal{Z}}^{\mathrm{cs}}(\text{Y},\mathbf{\Phi }))],$$

$$f^{\mathrm{cs}}:=-\frac{1}{L}{\mathbb{E}}_{\mathbf{\Phi },\text{Y}}[\ln ({\mathcal{Z}}^{\mathrm{cs}}(\text{Y},\mathbf{\Phi }))]$$

$$\mathrm{mmse}:=\frac{1}{L}{\mathbb{E}}_{\mathbf{\Phi },\text{S},\text{Y}}[\Vert \text{S}-\mathbb{E}[\text{X}|\text{Y},\mathbf{\Phi }]{\Vert }^2]=\frac{1}{L}{\mathbb{E}}_{\mathbf{\Phi },\text{S},\text{Z}}[\Vert \text{S}-\mathbb{E}[\text{X}|\mathbf{\Phi }\text{S}+\text{Z}\sqrt{\Delta },\mathbf{\Phi }]{\Vert }^2].$$

$$\mathrm{ymmse}:=\frac{1}{M}{\mathbb{E}}_{\mathbf{\Phi },\text{S},\text{Z}}[\Vert \mathbf{\Phi }(\text{S}-\mathbb{E}[\text{X}|\mathbf{\Phi }\text{S}+\text{Z}\sqrt{\Delta },\mathbf{\Phi }]){\Vert }^2]$$

$$\frac{d{i}^{\mathrm{cs}}}{d{\Delta }^{-1}}=\frac{\alpha B}{2}\mathrm{ymmse}.$$

$$\displaystyle{\rm ymmse}=\frac{{\rm mmse}}{1+{\rm mmse}/\Delta}+\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{$\scriptscriptstyle\mathcal{O}$}}_{L}(1).$$

$$\Sigma (E;\Delta {)}^{-2}$$

$$\psi (E;\Delta )$$

$$i(\tilde{\text{S}};\tilde{\text{Y}})$$

$$\underset{L\to \infty }{\lim }{i}^{\mathrm{cs}}=\underset{E\in [0,v]}{\min }{i}^{\mathrm{RS}}(E;\Delta ),$$

$$i^{\mathrm{RS}}(E;\Delta ):=\psi (E;\Delta )+i(\tilde{\text{S}};\tilde{\text{Y}}).$$

$$\tilde{E}(\Delta ):=\underset{E\in [0,v]}{\mathrm{argmin}}{i}^{\mathrm{RS}}(E;\Delta )$$

$$\text{z}{}^{(t)}$$

$$\hat{\text{s}}{}^{(t+1)}$$

$$\displaystyle[\eta({\textbf{y}};\Sigma^{2})]_{l}=\frac{\int d{\textbf{x}}_{l}\,{\textbf{x}}_{l}\,P_{0}({\textbf{x}}_{l})\exp\Big{(}-\frac{\|{\textbf{y}}_{l}-{\textbf{x}}_{l}\|^{2}}{2\Sigma^{2}}\Big{)}}{\int d{\textbf{x}}_{l}\,P_{0}({\textbf{x}}_{l})\exp\Big{(}-\frac{\|{\textbf{y}}_{l}-{\textbf{x}}_{l}\|^{2}}{2\Sigma^{2}}\Big{)}}=\frac{\sum_{k=1}^{K}{\textbf{a}}_{k}p_{k}\exp\Big{(}-\frac{\|{\textbf{y}}_{l}-{\textbf{a}}_{k}\|^{2}}{2\Sigma^{2}}\Big{)}}{\sum_{k=1}^{K}p_{k}\exp\Big{(}-\frac{\|{\textbf{y}}_{l}-{\textbf{a}}_{k}\|^{2}}{2\Sigma^{2}}\Big{)}}.$$

$$\displaystyle{\rm mmse}(\Sigma^{-2})\vcentcolon=\mathbb{E}_{\widetilde{\textbf{S}},\widetilde{\textbf{Z}}}\big{[}\|\widetilde{\textbf{S}}\!-\!\mathbb{E}[\widetilde{\textbf{S}}|\widetilde{\textbf{S}}\!+\!\widetilde{\textbf{Z}}\Sigma]\|^{2}\big{]}=\mathbb{E}_{\widetilde{\textbf{S}},\widetilde{\textbf{Z}}}\big{[}\|\widetilde{\textbf{S}}\!-\!\eta(\widetilde{\textbf{S}}\!+\!\widetilde{\textbf{Z}}\Sigma;\Sigma^{2})\|^{2}\big{]}.$$

$${\tau }_{t+1}^2=\frac{\Delta +\mathrm{mmse}({\tau }_t^{-2})}{\alpha B},t\ge 0\text{with}{\tau }_0^2=\frac{\Delta +v}{\alpha B}.$$

$$E^{(t)}:=\underset{L\to \infty }{\lim }\frac{1}{L}\Vert \text{s}-\hat{\text{s}}{}^{(t)}{\Vert }^2.$$

$$E^{(t+1)}=\mathrm{mmse}(\Sigma (E^{(t)};\Delta {)}^{-2}),$$

$$\underset{L\to \infty }{\lim }{i}^{\mathrm{cs}}\le \underset{E\in [0,v]}{\min }{i}^{\mathrm{RS}}(E;\Delta ).$$

$$\underset{L\to \infty }{\lim }{i}^{\mathrm{cs}}=\underset{E\in [0,v]}{\min }{i}^{\mathrm{RS}}(E;\Delta ).$$

$$\displaystyle{\rm ymmse}=\frac{{\rm mmse}}{1+{\rm mmse}/\Delta}+\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{$\scriptscriptstyle\mathcal{O}$}}_{L}(1).$$

$$\underset{L\to \infty }{\lim }\mathrm{mmse}$$

$$\underset{L\to \infty }{\lim }\mathrm{ymmse}$$

$$\underset{L\to \infty }{\lim }\mathrm{ymmse}=\frac{2}{\alpha B}\frac{d}{d{\Delta }^{-1}}\underset{L\to \infty }{\lim }{i}^{\mathrm{cs}}=\frac{2}{\alpha B}\frac{d}{d{\Delta }^{-1}}{i}^{\mathrm{RS}}(\tilde{E}(\Delta );\Delta ).$$

$${\mathrm{ymse}}_{\mathrm{AMP}}^{(t)}$$

Full text

In this work spatial coupling is used primarily as a proof technique. However our results also prove two important features of spatially coupled noisy linear random Gaussian estimation. First there is no algorithmically hard phase. This means that for such systems approximate message-passing always reaches the minimal-mean-square error. Secondly, in the limit of infinitely long coupled chain, the mutual information associated to spatially coupled systems is the same as the one of uncoupled linear random Gaussian estimation.

1. I Introduction

2. II Setting

3. II-A Gaussian random linear estimation

4. II-B Replica symmetric formula

5. II-C Approximate message-passing and state evolution

6. III Main Results

7. III-A Mutual information, MMSE and optimality of AMP

8. III-B The single first order phase transition scenario

9. IV Strategy of proof

10. IV-A A general interpolation

11. IV-B Various MMSE’s

12. IV-C The integration argument

13. IV-D Proof of $\Delta_{\rm Opt}\!=\!\Delta_{\rm RS}$ using spatial coupling

14. V Guerra’s interpolation method: Proof of Theorem III.1

15. VI Linking the measurement and standard MMSE: Proof of Lemma IV.6

16. VII Invariance of the mutual information: Proof of Theorem IV.9

17. VII-A Invariance of the mutual information between the CS and periodic SC models

18. VII-B Variation of MMSE profile

19. VII-C Invariance of the mutual information between the periodic and seeded SC systems

20. VIII Concentration properties

21. VIII-A Concentration of $\mathcal{L}$ on $\langle\mathcal{L}\rangle_{t,h}$

22. VIII-B Concentration of $\langle\mathcal{L}\rangle_{t,h}$ on $\mathbb{E}[\langle\mathcal{L}\rangle_{t,h}]$

23. VIII-C Concentration of ${\cal E}$ on $\mathbb{E}[\langle\mathcal{E}\rangle_{t,h}]$: Proof of Proposition VIII.1

24. IX MMSE relation for the CS model: Proof of Theorem III.4

25. IX-A Concentration of $\mathcal{Q}$ on $\mathbb{E}[\langle Q\rangle]$

26. IX-B Proof of the MMSE relation (26)

27. X Optimality of AMP

28. X-A MMSE relation for AMP

29. X-B Proof of the identity (210)

30. X-C Proof of Theorem III.7: Optimality of AMP

31. A I-MMSE formula

32. B Nishimori identities

33. C Differentiation of $i^{\rm RS}$ with respect to $\Delta^{-1}$

34. D The zero noise limit of the mutual information

35. E Concentration of the free energy: Proof of Proposition VIII.3

36. E-A Probabilistic tools

37. E-B Concentration of the free energy with respect to the Gaussian quenched variables

38. E-C Concentration of the free energy with respect to the signal

39. E-D Proof of Proposition VIII.3

40. F Super-additivity of the mutual information

41. G Unicity of $\Delta_{\rm RS}$ under the assumptions on $i_{\rm RS}(E;\Delta)$

I Introduction

Random linear projections and random matrices are ubiquitous in computer science and play an important role in machine learning, statistics and communications. In particular, the task of estimating a signal from linear random projections has a myriad of applications such as compressed sensing (CS) [1], code division multiple access (CDMA) in communications [2], error correction via sparse superposition codes [3], or Boolean group testing [4]. It is thus natural to ask what are the information theoretic limits for the estimation of a signal from the knowledge of its noisy random linear projections.

A particularly influential approach to this question has been through the use of the replica method of statistical physics [5], which allows to compute non rigorously the mutual information (MI) and the associated theoretically achievable minimal-mean-square error (MMSE). The replica method typically predicts the optimal performance through the solution of non-linear equations which interestingly coincide, for a range of parameters, with the predictions for the performance of a message-passing algorithm. In this context the algorithm is usually called approximate message-passing (AMP) [6, 7, 8].

In this contribution we prove rigorously that the replica formula for the MI is asymptotically exact in the case of random Gaussian linear projections. For example, our results put on a firm rigorous basis the Tanaka formula for CDMA [9] and allow to rigorously obtain the Bayesian MMSE in CS. In addition, we carefully discuss algorithmic consequences of our analysis, and prove that AMP reaches the MMSE for a large class of such problems except for a region called the hard phase. While AMP is an efficient low complexity algorithm, in the hard phase there is no known polynomial complexity local algorithm that allows to reach the MMSE, and it is believed that no such algorithms exist (hence the name “hard phase”).

Plenty of papers about structured linear problems make use of the replica method. In statistical physics, these date back to the late 80’s with the study of the perceptron and neural networks [10, 11, 12]. Of particular influence has been the work of Tanaka on CDMA [9] which has opened the way to a large set of contributions in information theory [13, 14]. In particular, the MI (or the free energy) in CS has been considered in a number of publications, e.g. [15, 16, 17, 18, 19, 7, 20, 8].

In a very interesting line of work, the replica formula has emerged following the study of the AMP algorithm. Again, the story of this algorithm is deeply rooted in statistical physics, with the work of Thouless, Anderson and Palmer [21] (thus the name “TAP” sometimes given to this approach). The earlier version, to the best of our knowledge, appeared in the late 80’s in the context of the perceptron problem [12]. For linear estimation, it was again developed initially in the context of CDMA [22]. It is, however, only after the application of this approach to CS [6] that the method has gained its current popularity. Of particular importance has been the development of the rigorous proof of state evolution (SE), an iterative equation that allows to track the performance of AMP, using techniques developed by [23] and [24]. Such techniques have their roots in the analysis of iterative forms of the TAP equations by Bolthausen [25]. Interestingly, the SE fixed points correspond to the extrema of the replica symmetric (RS) potential computed using the replica method, strongly hinting that AMP achieves the MMSE for many problems where it reaches the global minimum.

While our proof technique uses AMP and SE, it is based on two important additional ingredients. The first is the Guerra-Toninelli interpolation method [26, 27], that allows in particular to show that the RS potential yields an upper bound to the MI. This was already done for the CDMA problem in [27] (for binary signals) and here we extend this work to any discrete signal distribution. The converse requires more work and uses the second ingredient, namely a spatially coupled version of the model. In the context of CS such spatial coupling (SC) constructions were introduced in  [28, 29, 7, 8]. It was observed in [7, 8] and already proved in [29] that there is no hard phase for the AMP algorithm in the asymptotic regime of an infinite spatially coupled system. In the present paper spatial coupling is first used, not so much as an engineering construction, but as a mean to analyze the underlying uncoupled original system. Additionally, we also extend the results of [29] by proving that, not only does spatial coupling remove the existence of the hard phase at infinitesimal noise, but also provably leads to Bayes optimal results for any values of the signal-to-noise ratio and undersampling ratio. We use methods developed in the recent analysis of capacity-achieving spatially coupled low-density parity-check codes [30, 31, 32, 33] and sparse superposition codes [34, 35, 36].

We have recently applied a similar strategy to the factorization of low rank matrices [37, 38]. This, we believe, shows that the techniques and results developed in this paper are not only relevant for random linear estimation, but also in a broader context, and opens the way to prove many other results on estimation problems previously obtained with the heuristic replica method.

The techniques developed in this paper work well for the Bayes-optimal setting. One may wonder to what extent they can be applied away from this setting e.g., the LASSO or even L0 estimators. In the Bayes-optimal setting, the Bayes law implies simple but remarquable identities, often called Nishimori identities (see Appendix B), that prevent the phenomenon of replica symmetry breaking, and as a consequence the replica symmetric formula for the mutual information holds irrespective of the regime of parameters. In the absence of Nishimori identities our analysis breaks down. Another important feature of our analysis is the “full threshold saturation” of the message passing threshold to the information-theoretic one. For non-Bayesian problems threshold saturation may exist, but may only be partial (see e.g., [39]). A more in-depth study of these problems for random linear estimation away from the Bayesian-optimal setting is beyond the scope of this paper.

A summary of the present work has already appeared in [40]. The recent work [41, 42] also proves the replica formula for the MI in random linear estimation using a very different approach. In the present paper the replica formula is obtained under sligthly stronger hypothesis which could be improved to exactly match those of [42] at the expense of a few more technicalities. This is explained at the appropriate places.

II Setting

We use capital letters for random variables and small letters for particular realizations. Bold letters are vectors or matrices.

II-A Gaussian random linear estimation

In Gaussian random linear estimation one is interested in reconstructing a signal ${\textbf{s}}=(s_{i})_{i=1}^{N}\in\mathbb{R}^{N}$ from few noisy measurements ${\textbf{y}}=(y_{\mu})_{\mu=1}^{M}\!\in\!\mathbb{R}^{M}$ obtained from the projection of s by a random i.i.d Gaussian measurement matrix $\bm{\phi}=(\phi_{\mu i})_{\mu=1,i=1}^{M,N}\!\in\!\mathbb{R}^{M\times N}$. We study the problem in a probabilistic (Bayesian) setting, and therefore assume that the signal s is a particular realization of the random variable S (and similarly for the noise, measurement matrix etc). We consider i.i.d additive white Gaussian noise (AWGN) of known variance $\Delta$. Let the standardized noise components be $Z_{\mu}\sim\mathcal{N}(0,1)$, $\mu\in\{1,\dots,M\}$. Then the measurement model is ${\textbf{y}}=\bm{\phi}{\textbf{s}}+{\textbf{z}}\sqrt{\Delta}$, or equivalently

[TABLE]

The signal s may be structured in the sense that it is made of $L$ i.i.d $B$-dimensional sections ${\textbf{s}}_{l}\in\mathbb{R}^{B}$, $l\in\{1,\ldots,L\}$, distributed according to a discrete prior $P_{0}({\textbf{s}}_{l})=\sum_{k=1}^{K}p_{k}\delta({\textbf{s}}_{l}-{\textbf{a}}_{k})$ with a finite number $K$ of terms and all ${\textbf{a}}_{k}$’s with bounded components $\max_{k,j}|a_{kj}|\leq s_{\rm max}$ (here $1\leq k\leq K$, $1\leq j\leq B$). By a slight abuse of notation we denote ${\textbf{S}}\sim P_{0}$ to actually mean that each of the sections of S are drawn i.i.d from $P_{0}$, i.e. ${\textbf{S}}\sim P_{0}^{\otimes L}$. It is useful to keep in mind that ${\textbf{s}}=(s_{i})_{i=1}^{N}=({\textbf{s}}_{l})_{l=1}^{L}$ where ${\textbf{s}}_{l}\in\mathbb{R}^{B}$ and the total number of signal components is $N=LB$. The case $B=1$ corresponds to a structureless signal with purely scalar i.i.d components. We stress that $K$, $s_{\rm max}$ and $B$ are independent of $N,M,L$. We will refer to such priors simply as discrete priors, and to avoid extra technical complications we present all results and proofs in this context. We can extend some of our results to the case of priors that are mixtures of discrete and absolutely continuous parts and we briefly indicate how at appropriate places.

The matrix $\bm{\phi}$ has i.i.d Gaussian entries $\phi_{\mu i}\sim\mathcal{N}(0,1/L)$ (this scaling of the variance implies that the measurements have $\mathcal{O}(1)$ fluctuations). The measurement rate is $\alpha\vcentcolon=M/N$.

We borrow concepts from statistical mechanics and we often find it convenient to call the asymptotic large system size limit, where $N,M,L\to\infty$ with $\alpha$ and $B$ fixed, the “thermodynamic limit”. This regime where $\alpha$ is fixed is also sometimes referred to as the “high dimensional” regime in statistics.

The above setting is referred to as the CS model, and despite being more general than compressed sensing, we employ the vocabulary of this field.

The joint distribution of the signal and the measurements is equal to the AWGN channel transition probability $P^{\rm cs}({\textbf{y}}|{\textbf{s}},\bm{\phi})$ times the prior

[TABLE]

The information-theoretical optimal way of estimating the signal s follows from its posterior distribution. By the Bayes formula the posterior probability that the signal takes a value x given the knowledge of y and $\bm{\phi}$ is

[TABLE]

From now on we always denote by x a sample drawn according to the posterior distribution (3), X the associated random variable, and keep the notation s for the ground-truth signal to be inferred. The posterior normalization

[TABLE]

is also called the partition function. Note that it is related to the distribution of measurements through

[TABLE]

The MI (per section) $i^{\rm cs}\vcentcolon=I({\textbf{S}};{\textbf{Y}}|\bm{\Phi})/L$ is by definition

[TABLE]

where ${\textbf{S}}\sim P_{0}$. The last equality is obtained noticing that the conditional entropy of $P^{\rm cs}({\textbf{y}}|{\textbf{s}},\bm{\phi})$ is simply the differential entropy of i.i.d Gaussian random variables of variance $\Delta$, that is h({\textbf{Y}}|{\textbf{S}},\bm{\Phi})=(M/2)\big{(}\ln(2\pi\Delta)+1\big{)}. Note that up to an additive constant the MI is equal to

[TABLE]

which is the average free energy (per section) in statistical physics. We will consider at some point the free energy for a given measurement and measurment matrix realization defined as $f^{\rm cs}({\textbf{y}},\bm{\phi})\vcentcolon=-\ln(\mathcal{Z}^{\rm cs}({\textbf{y}},\bm{\phi}))/L$ and show that it concentrates.

The usual MMSE estimator which minimises the mean-square error is $\mathbb{E}[{\textbf{X}}|{\textbf{y}},\bm{\phi}]$, which is the mean of the posterior (3) (recall that we use s for the ground-truth value of the signal, ${\textbf{S}}\sim P_{0}$ the associated random variable, and ${\textbf{X}}\sim P^{\rm cs}(\cdot|{\textbf{y}},\bm{\phi})$ for a random variable drawn according to the posterior), and the MMSE per section is

[TABLE]

Unfortunately, this quantity is rather difficult to access directly from the MI. For this reason, it is more convenient to consider the measurement MMSE defined as

[TABLE]

which is directly related to the MI by an I-MMSE relation [43]:

[TABLE]

We verify this relation for the present setting by explicit algebra in appendix A.

We will prove in sec. IX the following non-trivial relation between the MMSE’s for almost every (a.e.) $\Delta$:

[TABLE]

In this paper we always denote by \mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}_{L}(1) a quantity such that \lim_{L\to\infty}\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}_{L}(1)=0 and that might depend on the various fixed problem parameters such as $(P_{0},B,\Delta,\alpha)$ etc.

Thus, if we can compute the MI, we can compute the measurement MMSE and conversely. Moreover from the measurement MMSE we get the usual MMSE and conversely.

II-B Replica symmetric formula

Define $v\vcentcolon=\mathbb{E}[\|{\textbf{S}}\|^{2}]/L=\sum_{k=1}^{K}p_{k}\|{\textbf{a}}_{k}\|^{2}$ and for $0\leq E\leq v$,

[TABLE]

Let $i(\widetilde{\textbf{S}};\widetilde{\textbf{Y}})$ be the MI for a $B$-dimensional denoising model $\widetilde{\textbf{y}}=\widetilde{\textbf{s}}+\widetilde{\textbf{z}}\,\Sigma$ with $\widetilde{\textbf{s}}$, $\widetilde{\textbf{z}}$, $\widetilde{\textbf{y}}$ all in $\mathbb{R}^{B}$ and $\widetilde{\textbf{S}}\sim P_{0}$, $\widetilde{\textbf{Z}}\sim\mathcal{N}(0,\mathbf{I}_{B})$, $\mathbf{I}_{B}$ the $B$-dimensional identity matrix (recall capital letters are for random variables, small ones for realizations). A straightforward exercise leads to

[TABLE]

where $\widetilde{\textbf{X}}\sim P_{0}$.

The replica method yields the replica symmetric (RS) formula for the MI of model (1),

[TABLE]

where the RS potential is

[TABLE]

This formula was first derived by Tanaka [9] for binary signals using the heuristic replica method, and later on for different and more general settings in [15, 7, 8, 20, 44, 17, 16] still using the replica method. See also [45, 46] for comparisons between rigorous bounds and replica predictions. One of our main contribution is to prove that the replica predictions is indeed correct, at least in some settings that will precisely describe in sec. III.

In the following we will denote

[TABLE]

when it is unique (this is the case except at isolated first order phase transition points). In order to alleviate the subsequent notations we often do not explicitly write the $\Delta$ dependence of $\widetilde{E}$ when the context is clear.

We consider models with a discrete prior $P_{0}$ such that (s.t) (15) has at most three stationary points (see the discussion in sec. III-B). This implies that $i^{\rm RS}(\widetilde{E}(\Delta);\Delta)$ has at most one non-analyticity point denoted $\Delta_{\rm RS}$ which is precisely the point where the ${\rm argmin}$ in (16) is non-unique. A complete proof of this fact is a lengthy but elementary exercise in real analysis and we give the main thrust of the argument in Appendix G. When $i^{\rm RS}(\widetilde{E}(\Delta);\Delta)$ is analytic over $\mathbb{R}^{+}$ we simply set $\Delta_{\rm RS}\!=\!\infty$.

The most common non-analyticity in this context is a non-differentiability point of $i^{\rm RS}(\widetilde{E}(\Delta);\Delta)$. By virtue of (9) and (10) this corresponds to a jump discontinuity of the MMSE’s, and one speaks of a first order phase transition. Another possibility is a discontinuity in higher derivatives of the MI, in which case the MMSE’s are continuous but non differentiable and one speaks of higher order phase transitions.

For the reader interested in visualizing the potential shape in various settings of $(P_{0},\alpha,\Delta,B)$ we refer to [7, 8] (for $B=1$) or [35, 47] (for $B\geq 2$) where many examples can be found.

II-C Approximate message-passing and state evolution

II-C1 Approximate message-passing algorithm

Define the following rescaled variables $\bm{\phi}_{0}\vcentcolon=\bm{\phi}/\sqrt{\alpha B}$, ${\textbf{y}}_{0}\vcentcolon={\textbf{y}}/\sqrt{\alpha B}$. These definitions are useful in order to be coherent with the definitions of [6, 23] in order to apply directly their theorems. The AMP algorithm constructs a sequence of estimates $\widehat{{\textbf{s}}}\!\!~{}^{(t)}\in\mathbb{R}^{N}$ and “residuals” ${\textbf{z}}\!\!~{}^{(t)}\in\mathbb{R}^{M}$ (these play the role of an effective noise not to be confused with z) according to the following iterations

[TABLE]

with initialization $\widehat{{\textbf{s}}}\!\!~{}^{(0)}=0$ (any quantity with negative time index is also set to the zero vector). In the Bayesian optimal setting, the section wise denoiser $\eta({\textbf{y}};\Sigma^{2})$ (which returns a vector with same dimension as its first argument) is the MMSE estimator associated to an effective AWGN channel ${\textbf{y}}={\textbf{x}}+{\textbf{z}}\Sigma$, ${\textbf{x}},{\textbf{y}},{\textbf{z}}\in\mathbb{R}^{N}$, ${\textbf{Z}}\sim\mathcal{N}(0,\textbf{I}_{N})$. The $l$-th section of the $N$-dimensional vector $\eta({\textbf{y}};\Sigma^{2})$ is a $B$-dimensional vector given by

[TABLE]

The last form is obtained using the explicit form of the discrete prior. In (17) $[\eta^{\prime}({\textbf{y}};\Sigma^{2})]_{i}$ denotes the $i$-th scalar component of the gradient of $\eta$ w.r.t its first argument. In order to define $\tau_{t}$ we need the following function. Define the ${\rm mmse}$ function associated to the $B$-dimensional denoising model $\widetilde{\textbf{y}}=\widetilde{\textbf{s}}+\widetilde{\textbf{z}}\,\Sigma$ introduced previously as

[TABLE]

Then $\tau_{t}$ is a sequence of effective AWGN variances precomputed by the following recursion:

[TABLE]

II-C2 State evolution

The asymptotic performance of AMP for the CS model can be rigorously tracked by state evolution in the scalar $B\!=\!1$ case [23, 29]. The vectorial $B\!\geq\!2$ case requires extending the rigorous analysis of SE, which at the moment has not been done to the best of our knowledge. Nevertheless, we conjecture that SE (see (23) below) tracks AMP for any $B$. This is numerically confirmed in [35] and proven for power allocated sparse superposition codes [48] (these correspond to a special vectorial case with $B\geq 2$).

Denote the asymptotic MSE per section obtained by AMP at iteration $t$ as

[TABLE]

The SE recursion tracking the performance of AMP is

[TABLE]

with initialization $E^{(0)}=v$, that is without any knowledge about the signal other than its prior distribution. Monotonicity properties of the mmse function (20) imply that $E^{(t)}$ is a decreasing sequence s.t $\lim_{t\to\infty}E^{(t)}\!=\!E^{(\infty)}$ exists, see [36] for the proof of this fact.

II-C3 Algorithmic threshold and link with the potential

Let us give a natural definition for the AMP threshold.

Definition II.1** (AMP algorithmic threshold)**

$\Delta_{\rm AMP}$* is the supremum of all $\Delta$ s.t the SE fixed point equation $E={\rm mmse}(\Sigma(E;\Delta)^{-2})$ has a unique solution for all noise values in $[0,\Delta]$.*

Remark II.2** (SE and $i^{\rm RS}$ link)**

It is easy to prove by simple algebra that if $E$ is a fixed point of the SE recursion (23), then it is an extremum of (15), that is the fixed point equation $E={\rm mmse}(\Sigma(E;\Delta)^{-2})$ implies $\partial i^{\rm RS}(E;\Delta)/\partial E=0$. Therefore $\Delta_{\rm AMP}$ is also the smallest solution of $\partial i^{\rm RS}/\partial E\!=\!\partial^{2}i^{\rm RS}/\partial E^{2}\!=\!0$; in other words it is the “first” horizontal inflexion point appearing in $i^{\rm RS}(E;\Delta)$ when $\Delta$ increases. In particular $\Delta_{\rm AMP}\leq\Delta_{\rm RS}$ because $\Delta_{\rm RS}$ is precisely the point where the argmin in (16) is not unique.

III Main Results

III-A Mutual information, MMSE and optimality of AMP

III-A1 Mutual information and information theoretic threshold

Our first result states that the minimum of the RS potential (15) upper bounds the asymptotic MI.

Theorem III.1** (Tight upper bound)**

For model (1) with any $B$ and discrete prior $P_{0}$,

[TABLE]

This result generalizes the one already obtained for CDMA in [27]. One of our main result is the equality in the scalar case $B=1$, namely the proof of the RS formula derived previously in [9, 15, 7, 8, 44] based on the heuristic replica method from statistical physics.

Theorem III.2** (RS formula for $i^{\rm cs}$)**

Take $B=1$ and assume $P_{0}$ is a discrete prior s.t the RS potential $i^{\rm RS}(E;\Delta)$ in (15) has at most three stationary points (as a function of $E$). Then for any $\Delta$ the RS formula is exact, that is

[TABLE]

It is conceptually useful to define the following threshold.

Definition III.3** (Information theoretic (or optimal) threshold)**

Define $\Delta_{\rm Opt}\vcentcolon=\sup\{\Delta\ \text{s.t}\ \lim_{L\to\infty}i^{\rm cs}$ is analytic in $]0,\Delta[\}$.

This is also one of the most fundamental definitions of a (static) phase transition threshold and also plays an important role in our analysis. Theorem III.2 gives us an explicit formula to compute the information theoretic threshold, namely $\Delta_{\rm Opt}=\Delta_{\rm RS}$. Notice that we have not assumed anything on the number of non-analyticity points (phase transitions) of $\lim_{L\to\infty}i^{\rm cs}$.

III-A2 Minimal mean-square errors

An important result is the relation between the measurement MMSE and usual MMSE (proven in sec. IX). The next theorem is independent of the previous ones. In particular the proof does not rely on SE which allows to relax the constraint $B=1$ (for which SE is known to rigorously track AMP).

Theorem III.4** (MMSE relation)**

For model (1) with any (finite) $B$, any discrete prior $P_{0}$ and for a.e. $\Delta$, the usual and measurement MMSE’s given by (7) and (8) are related by

[TABLE]

Corollary III.5** (MMSE and measurement MMSE)**

Under the same assumptions as in Theorem III.2 and for any $\Delta\!\neq\!\Delta_{\rm RS}$, the usual and measurement MMSE given by (7) and (8) satisfy

[TABLE]

where $\widetilde{E}(\Delta)$ is the unique global minimum of $i^{\rm RS}(E;\Delta)$ for $\Delta\neq\Delta_{\rm RS}$.

Proof:

The proof follows from standard arguments that we immediately sketch here. We first remark that the sequence $i^{\rm cs}$ (w.r.t $L$) is concave in $\Delta^{-1}$ and the thermodynamic limit $\lim_{L\to\infty}i^{\rm cs}$ exists. Concavity is intuitively clear from the I-MMSE relation (9) which implies that $di^{\rm cs}/d\Delta^{-1}$ is non-decreasing in $\Delta^{-1}$. A detailed proof of concavity that applies in the present context is found in [49]. The proof of existence of the limit in [27] for a binary distribution $P_{0}$ directly extends to the more general setting here. Alternatively an interpolation method similar to that of sec. VII shows that the sequence is super-additive which implies existence of the thermodynamic limit (by Fekete’s lemma). By a standard theorem of real analysis the limit of a sequence of concave functions is: $i)$ concave and continuous on any compact subset, $ii)$ differentiable almost everywhere, $iii)$ the limit and derivative can be exchanged at every differentiability point. Here we know from Theorem III.2 that the only non-differentiability point of $\lim_{L\to\infty}i^{\rm cs}$ is $\Delta_{\rm RS}$. Therefore thanks to the I-MMSE relation (9) we deduce that for $\Delta\neq\Delta_{\rm RS}$,

[TABLE]

The explicit calculation of the total derivative is found in appendix C and yields relation (28). Finally (27) follows from (26) in Theorem III.4 and (28). ∎

The proofs of Theorems III.1 and III.2 are presented in sec. IV and V. We conjecture that Theorem III.2 and Corollary III.5 hold for any $B$. Their proofs require a control of AMP by SE, a result that to the best of our knowledge is currently available in the literature only for $B=1$. Proving SE for all $B$ would imply that all the results of this paper are valid for the structured case, and we believe that this is not out of reach111Since completion of the first version of this manuscript some works related to the extension of the state evolution analysis to more general settings, including structured priors, have appeared [50, 51]. It is probable that these combined with the present analysis allow to generalize our results to the $B\geq 2$ case. We tend to believe that it is the case but leave this question open for future work..

Remark III.6** (More general priors and phase transition scenarios)**

In [42] priors are more general in two respects. First they can have mixtures of discrete and continuous parts. Second the RS potential $i^{\rm RS}(E;\Delta)$ may have more than three stationary points but with at most two that become global minima. This second point means that the single first order phase transition scenario discussed below (see paragraph III-B) is the same. All our proofs and statements also hold in this case but we have formulated them in a slightly more restrictive way just for clarity of the general picture. Concerning the first point one can presumably directly extend Theorems  III.1 and III.2 (and hence identity (28)) to priors with continuous and discrete parts. The idea is to use limiting arguments, approximating a general prior by a sequence of discrete ones, and taking the limit of (24) and (25). In this process one must make sure that one stays within the class of priors such that the RS potential $i^{\rm RS}(E;\Delta)$ has at most three stationary points (or more but with at most two global minima) and it is possible to do so as long as the limiting prior also satisfies this condition.

III-A3 Optimality of approximate message-passing in Gaussian random linear estimation

Let us now give the results related to the optimality of AMP for the CS model (1). These are proven in sec. X. Let the measurement MSE of AMP be

[TABLE]

Moreover, recall the definition of the asymptotic MSE of AMP $E^{(t)}$ given in (22).

Theorem III.7** (Optimality of AMP)**

Under the same assumptions as in Theorem III.2 and if $\Delta<\Delta_{\rm AMP}$ or $\Delta>\Delta_{\rm RS}$, then AMP is almost surely optimal in the following sense:

[TABLE]

where ${\rm mmse}$ and ${\rm ymmse}$ are defined by (7) and (8).

III-A4 Results on spatially coupled Gaussian random linear estimation

Another fundamental result, that is key in our proof of the previous theorems but is in itself of independent interest, is an equivalence theorem (see Theorem IV.9). It roughly states that in the limit of infinit spatially coupled chain, the MI of the CS model (1) and the spatially coupled CS models are equal. Informally speaking: denote $i_{\Gamma,w}^{\rm seed}$ and $i_{\Gamma,w}^{\rm per}$ the MI for two different spatially coupled models, where the former is associated to an open coupled chain, the latter to a closed periodic chain. Then the following theorem states that these coupled systems have same MI as the plain model when the number of sections $L$ diverges, and, for the open chain model, when then the number of sub-systems $\Gamma$ in the coupled chain also diverges afterwards. We already state this result and an important corollary here, for the convenience of the reader, but refer to sec. IV-D for the precise definitions of the spatially coupled CS models.

Theorem III.8** (Invariance of the MI)**

The following MI limits exist and are equal for any (odd) $\Gamma$ and $w\!\in\!\{0,\ldots,(\Gamma\!-\!1)/2\}$:

[TABLE]

Moreover for any fixed $w$, we also have

[TABLE]

Let us mention that the proof of this result is based on a crucial new interpolation technique developed specifically for this purpose, and called sub-extensive interpolation, which we believe is of independent interest222Since completion of the first version of this manuscript the sub-extensive interpolation method has been further developed [52]. It then evolved into another type of interpolation technique called “adaptive interpolation method”, see [53, 54, 55].. The high-level idea is to compare two statistical models by interpolating one onto the other. But, as oposed to the canonical Guerra-Toninelli interpolation [56] where this is done “all at once”, in the sub-extensive interpolation method this is done in small increments, which permits a more refined control along the interpolation path. We refer to sec. VII for a precise explanation of the method.

In addition, as a corollary of the exactness of state evolution for the SC models [23, 29], the threshold saturation Lemma IV.8 and of Corollary III.5 we obtain the following optimality result of AMP for spatially coupled Gaussian random linear estimation. This result states that the mean square-error of AMP in reconstructing the signal in the spatially coupled system, $E^{c,(t)}$, as well as its mean square-error ${\rm ymse}_{\rm AMP}^{{\rm c},(t)}$ in reconstructing the linear projections $\bm{\phi}{\textbf{s}}$ both tend to the optimal values as the number of iterations diverges, and when the noise does not exceed the information theoretic threshold.

Theorem III.9** (Optimality of AMP for spatially coupled estimation)**

Under the same assumptions as in Theorem III.2 and if $\Delta<\Delta_{\rm Opt}=\Delta_{\rm RS}$, then AMP is almost surely optimal for spatially coupled Gaussian random linear estimation in the following sense:

[TABLE]

where the asymptotic MSE $E^{{\rm c},(t)}$ and measurement MSE ${\rm ymse}_{\rm AMP}^{{\rm c},(t)}$ of AMP for the spatially coupled models are defined similarly as (22) and (30), respectively, but with a spatially coupled measurement matrix $\bm{\phi}$ and where ${\rm mmse}$ and ${\rm ymmse}$ are defined by (7) and (8).

Note that this corollary is not explicitly used in this paper and comes as a bonus. In [29], a similar argument was used to prove that, with vanishing noise, spatial coupling allowed for an optimal reconstruction as long as the undersampling rate (or measurement rate) exceeds the upper Rényi information dimension of the signal. This result allows for a wide set of signal prior distributions involving discrete as well as continuous parts. Our analysis is restricted to the case of discrete priors, but in this case theorem III.9 generalizes the result of [29]. Informally, it simply states that spatial coupling always allows to reach Bayes optimal performances.

III-B The single first order phase transition scenario

In this contribution, we assume that $P_{0}$ is discrete and s.t (15) has at most three stationary points. Let us briefly discuss what this hypothesis entails.

Three scenarios are possible: $\Delta_{\rm AMP}\!<\!\Delta_{\rm RS}$ (one first order phase transition); $\Delta_{\rm AMP}\!=\!\Delta_{\rm RS}\!<\!\infty$ (one higher order phase transition); $\Delta_{\rm AMP}\!=\!\Delta_{\rm RS}\!=\!\infty$ (no phase transition). Here we will consider the most interesting (and challenging) first order phase transition case where a gap between the algorithmic AMP and information theoretic performance appears. The cases of no or higher order phase transition, which present no algorithmic gap, follow as special cases from our proof. It should be noted that in these two cases spatial coupling is not really needed and the proof may be achieved by an “area theorem” as already argued in [57].

Recall the notation $\widetilde{E}(\Delta)\!=\!{\rm argmin}_{E\in[0,v]}i^{\rm RS}(E;\Delta)$. At $\Delta_{\rm RS}$, when the ${\rm argmin}$ is a set with two elements, one can think of $\widetilde{E}(\Delta)$ as a discontinuous function.

The picture for the stationary points of (15) is as follows. For $\Delta\!<\!\Delta_{\rm AMP}$ there is a unique stationary point which is a global minimum $\widetilde{E}(\Delta)$ and we have $\widetilde{E}(\Delta)\!=\!E^{(\infty)}$, the fixed point of SE (23). At $\Delta_{\rm AMP}$ the function $i^{\rm RS}$ develops a horizontal inflexion point, and for $\Delta_{\rm AMP}\!<\!\Delta\!<\!\Delta_{\rm RS}$ there are three stationary points: a local minimum corresponding to $E^{(\infty)}$, a local maximum, and the global minimum $\widetilde{E}(\Delta)$. It is not difficult to argue that $\widetilde{E}(\Delta)\!<\!E^{(\infty)}$ in the interval $\Delta_{\rm AMP}\!<\!\Delta\!<\!\Delta_{\rm RS}$. At $\Delta_{\rm RS}$ the local and global minima switch roles, so at this point the global minimum $\widetilde{E}(\Delta)$ has a jump discontinuity. For all $\Delta\!>\!\Delta_{\rm RS}$ there is at least one stationary point which is the global minimum $\widetilde{E}(\Delta)$ and $\widetilde{E}(\Delta)\!=\!E^{(\infty)}$ (the other stationary points can merge and annihilate each other as $\Delta$ increases).

Finally we note that with the help of the implicit function theorem for real analytic functions, we can show that $\widetilde{E}(\Delta)$ is an analytic function of $\Delta$ except at $\Delta_{\rm RS}$. Therefore $i^{\rm RS}(\widetilde{E}(\Delta),\Delta)$ is analytic in $\Delta$ except at $\Delta_{\rm RS}$.

Let us mention that references [41, 42] require a weaker “single-crossing” assumption that includes the scenarios covered by our proof333One example of prior where our assumption of at most three stationary points is not verified is a uniform $P_{0}$ supported on $\{0,1,10\}$ (for a measurement rate $\alpha$ low enough). We thank a reviewer for pointing out this explicit example.. We believe that our proof technique may be extended to include additional scenarios with more generic priors at the cost of additional technicalities.

IV Strategy of proof

Let us start with a word about subsequent notations used. It is useful to distinguish two types of expectations. The first one are expectations w.r.t posterior distributions, e.g., the expectation $\mathbb{E}_{{\textbf{X}}|{\textbf{y}},\bm{\phi}}$ w.r.t (3), which we will most of the time denote as Gibbs averages $\langle\!-\!\rangle$. The second one are the expectations w.r.t all so-called quenched variables, e.g., $\bm{\Phi}$, S, Z, which will be denoted by $\mathbb{E}$. Subscripts in these expectations will be explicitly written down only when necessary to avoid confusions. For example the MMSE estimator becomes with these notations $\mathbb{E}[{\textbf{X}}|{\textbf{y}},\bm{\phi}]=\langle{\textbf{X}}\rangle$ and the mmse (7) is simply ${\rm mmse}=\mathbb{E}[\|{\textbf{S}}\!-\!\langle{\textbf{X}}\rangle]\|^{2}]/L$. For the measurement MMSE (8) we have ${\rm ymmse}=\mathbb{E}[\|\bm{\Phi}({\textbf{S}}\!-\!\langle{\textbf{X}}\rangle)\|^{2}]/M$.

IV-A A general interpolation

We have already seen in sec. II-B that the RS potential (15) involves the MI of a denoising model. One of the main tools that we use is an interpolation between this denoising model and the original CS model (1) (the denoising model here is the same up to its dimensionality, thus we use the same notation). Consider a set of observations $[{\textbf{y}},\widetilde{\textbf{y}}]$ from the following channels

[TABLE]

where ${\textbf{Z}}\sim\mathcal{N}(0,\mathbf{I}_{M})$, $\widetilde{\textbf{Z}}\sim\mathcal{N}(0,\mathbf{I}_{N})$, $t\in[0,1]$ is the interpolating parameter and the “signal-to-noise functions” $\gamma(t)$ and $\lambda(t)$ satisfy the constraint

[TABLE]

with the following boundary conditions

[TABLE]

We also require $\gamma(t)$ to be strictly increasing. Notice that (38) implies

[TABLE]

so that $\lambda(t)$ is strictly decreasing with $t$.

In order to prove concentration properties that are needed in our proofs, we will actually work with a more complicated perturbed interpolated model where we add a set of extra observations that come from another “side channel” denoising model

[TABLE]

$\widehat{\textbf{Z}}\sim\mathcal{N}(0,\mathbf{I}_{N})$. Here the snr $h$ is “small” and one should keep in mind that it will be removed in the process of the proof, i.e. $h\!\to\!0_{+}$.

Define $\mathring{{\textbf{y}}}\!\vcentcolon=\![{\textbf{y}},\widetilde{\textbf{y}},\widehat{\textbf{y}},\bm{\phi}]$ as the concatanation of all observations and the measurement matrix. Moreover, it will be useful to use the following notation: $\bar{\textbf{x}}\vcentcolon={\textbf{x}}-{\textbf{s}}$. In particular note $[\bm{\phi}\bar{\textbf{x}}]_{\mu}=\sum_{i=1}^{N}\phi_{\mu i}(x_{i}-s_{i})$. Our central object of study is the posterior of the general perturbed interpolated model:

[TABLE]

where the Hamiltonian is

[TABLE]

and the partition function

[TABLE]

We replaced y, $\widetilde{\textbf{y}}$, $\widehat{\textbf{y}}$ by their expressions (37) and (41). Expectations w.r.t the Gibbs measure (42) are denoted $\langle\!-\!\rangle_{t,h}$, namely

[TABLE]

and expectations w.r.t all quenched random variables $\bm{\phi}$, s, y, $\widetilde{\textbf{y}}$, $\widehat{\textbf{y}}$ (or equivalently the independent $\bm{\phi}$, s, z, $\widetilde{\textbf{z}}$, $\widehat{\textbf{z}}$) by $\mathbb{E}=\mathbb{E}_{\bm{\Phi}}\mathbb{E}_{{\textbf{S}}}\mathbb{E}_{{\textbf{Y}}|\bm{\Phi},{\textbf{S}}}\mathbb{E}_{\widetilde{\textbf{Y}}|{\textbf{S}}}\mathbb{E}_{\widehat{\textbf{Y}}|{\textbf{S}}}=\mathbb{E}_{\bm{\Phi}}\mathbb{E}_{{\textbf{S}}}\mathbb{E}_{{\textbf{Z}}}\mathbb{E}_{\widetilde{\textbf{Z}}}\mathbb{E}_{\widehat{\textbf{Z}}}$. The last term appearing in the Hamiltonian does not depend on x and cancels in the posterior. The reason for adding this term is purely technical: it makes an additive contribution to the MI and free energy, which makes them concave in $h$.

The MI for the perturbed interpolated model is defined similarly as (II-A) and one obtains

[TABLE]

Note that $i_{1,0}=i^{\rm cs}$ given by (II-A). It is also useful to define the free energy per section for a given realisation of quenched variables

[TABLE]

We immediately prove an easy but useful lemma.

Lemma IV.1** (Concavity in $h$ of the mutual information)**

The MI for the perturbed interpolated model $i_{t,h}$ is concave in $h$ for all $t$. The same is true for the free energy $f_{t,h}(\mathring{{\textbf{y}}})$.

Proof:

Let

[TABLE]

One can compute the first derivative of $i_{t,h}$ and finds

[TABLE]

because, recalling the definition of the Hamiltonian (43), we have

[TABLE]

and $S_{i}$ is independent of the zero mean Gaussian variable $\widehat{Z}_{i}$. We now compute the second derivative:

[TABLE]

In order to obtain (48), note that

[TABLE]

We also have

[TABLE]

Therefore

[TABLE]

Combining everything gives (48). We observe that the second derivative is non-positive since $|\langle X_{i}\rangle_{t,h}|\!\leq\!s_{\rm max}$ and therefore $i_{t,h}$ is concave in $h$. The second derivative of the free energy $d^{2}f_{t,h}(\mathring{{\textbf{y}}})/dh^{2}$ is given by the right hand side of (48) but without the expectation $\mathbb{E}$. Therefore $f_{t,h}(\mathring{{\textbf{y}}})$ is also concave in $h$. ∎

Remark IV.2** (Thermodynamic limit)**

The interpolation methods used in this paper imply super-additivity of the mutual information and thus (by Fekete’s lemma) the existence of the thermodynamic limit $\lim_{L\to\infty}i_{t,h}$. The reader is refered to paragraph VII-A6 in Sec. VII for a discussion of this point. Concavity in $h$ thus implies that the convergence of the sequence $i_{t,h}$ is uniform on all $h$-compact subsets and therefore $\lim_{h\to 0}\lim_{L\to\infty}i_{t,h}=\lim_{L\to\infty}\lim_{h\to 0}i_{t,h}$ (note that the MI is bounded for any $h$, so $h=0$ can be included in the compact subset). This property will be used later on.

Remark IV.3** (Interpretation of (38))**

Constraint (38), or snr conservation, is essential. It expresses that as $t$ decreases from $1$ to [math], we slowly decrease the snr of the CS measurements and make up for it in the denoising model. When $t\!=\!0$ the snr vanishes for the CS model, and no information is available about s from the compressed measurements, information comes only from the denoising model. Instead at $t\!=\!1$ the noise is infinite in the denoising model and letting also $h\!\to\!0$ we recover the CS model (1). Let us further interpret (38). Given a CS model of snr $\Delta^{-1}$, by remark II.2 and (23), the global minimum of (15) is the MMSE of an “effective” denoising model of snr $\Sigma(E;\Delta)^{-2}$. Therefore, the interpolated model (43) (at $h\!=\!0$) is asymptotically equivalent (in the sense that it has the same MMSE) to two independent denoising models: an “effective” one of snr $\Sigma(E;\gamma(t)^{-1})^{-2}$ associated to the CS model, and another one with snr $\lambda(t)$. Proving Theorem III.2 requires the interpolated model to be designed s.t its MMSE equals the MMSE of the CS model (1) for almost all $t$. Knowing that the estimation of s in the interpolated model comes from independent channels, this MMSE constraint induces (38).

Remark IV.4** (Nishimori identity)**

We place ourselves in the Bayes optimal setting which means that $P_{0},\Delta,\gamma(t)$, $\lambda(t)$ and $h$ are known. The perturbed interpolated model is carefully designed so that each of the three x-dependent terms in (43) corresponds to a “physical” channel model with a properly defined transition probability. As a consequence the Nishimori identity holds. This remarkable and general identity that follows from the Bayes formula plays an important role in our calculations. For any (integrable) function $g({\textbf{x}},{\textbf{s}})$ where s is the signal, we have

[TABLE]

where X and ${\textbf{X}}^{\prime}$ are i.i.d vectors, called “replicas”, distributed according to the product measure associated to (42), namely the measure $P_{t,h}({\textbf{x}}|\mathring{{\textbf{y}}})P_{t,h}({\textbf{x}}^{\prime}|\mathring{{\textbf{y}}})$. We slightly abuse notation here by denoting the expectation operator w.r.t the posterior measure for X and this product measure for ${\textbf{X}},{\textbf{X}}^{\prime}$ with the same bracket $\langle\!-\!\rangle_{t,h}$:

[TABLE]

See appendix B for a derivation of the basic identity (49) as well as many other useful consequences.

IV-B Various MMSE’s

We will need the following I-MMSE lemma that straightforwardly extends to the perturbed interpolated model the usual I-MMSE relation (9) for the AWGN channel. The proof, for the simpler CS model, is found in appendix A but it remains valid here as well because the Nishimori identity (49) holds for the perturbed interpolated model. Let

[TABLE]

Lemma IV.5** (I-MMSE)**

The perturbed interpolated model at $t=1$ verifies the following I-MMSE relation:

[TABLE]

Let us give a useful link between ${\rm ymmse}_{t,h}$ and the usual MMSE for this model,

[TABLE]

For the perturbed interpolated model the following holds (see sec. VI for the proof).

Lemma IV.6** (MMSE relation)**

For a.e. $h$,

[TABLE]

In this lemma \lim_{L\to\infty}\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}_{L}(1)=0 but in our proof \mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}_{L}(1) is not uniform in $h$ and diverges like $h^{-1/2}$ as $h\!\to\!0$. For this reason we cannot interchange the limits $L\!\to\!\infty$ and $h\!\to\!0$ in (53). This is not only a technicality, because in the presence of a first order phase transition one has to somehow deal with the discontinuity in the MMSE.

IV-C The integration argument

IV-C1 Sub-optimality inequality

The AMP algorithm is sub-optimal and this can be expressed as a useful inequality. When used for inference over the CS model (1) (i.e. over the perturbed interpolated model with $t\!=\!1,h\!=\!0$) one gets $\limsup_{L\to\infty}E_{1,0}\leq E^{(\infty)}$. Adding new measurements coming from a side channel can only improve optimal inference thus $E_{1,h}\leq E_{1,0}$ so that $\limsup_{L\to\infty}E_{1,h}\leq E^{(\infty)}$. Combining this with Lemma IV.6 and using that $E(1+E/\Delta)^{-1}$ is an increasing function of $E$, one gets for a.e. $h$

[TABLE]

We note that we could use a version of Theorem III.4 valid for a.e. $\Delta$ to get the same inequality for $h=0$ and a.e. $\Delta$. This does not make any major difference nor simplification in the subsequent argument so we prefer to use the weaker Lemma IV.6 at this point.

IV-C2 **Below the algorithmic threshold

$\Delta<\Delta_{\rm AMP}$**

In this noise regime $E^{(\infty)}\!=\!\widetilde{E}$ the global minimum of $i^{\rm RS}$ (recall (16) and remark II.2) so we replace $E^{(\infty)}$ by $\widetilde{E}$ in the r.h.s of (54). An elementary calculation done in appendix C shows that

[TABLE]

Using this identity and Lemma IV.5, the inequality (54) becomes

[TABLE]

or equivalently

[TABLE]

Integrating the last inequality over $[0,\Delta]\subset[0,\Delta_{\rm AMP}]$ and using Fatou’s lemma we get for a.e. $h$

[TABLE]

By remark IV.2 we can replace $\liminf$ by $\lim$ in (58), take the limit $h\!\to\!0$ and permute the limits. This yields

[TABLE]

In the noiseless case $\Delta\!=\!0$ we have from the replica potential $i^{\rm RS}(\widetilde{E};0)=H({\textbf{S}})$ and also $\lim_{L\to\infty}i^{\rm cs}|_{\Delta=0}=H({\textbf{S}})$, with $H({\textbf{S}})$ the Shannon entropy of ${\textbf{S}}\!\sim\!P_{0}$. We stress that these two statements are true irrespective of $\alpha$ and $\rho$ because the alphabet is discrete. A justification is found in appendix D. For a mixture of continuous and discrete alphabet we still have $i^{\rm RS}(\widetilde{E};0)=\lim_{L\to\infty}i^{\rm cs}|_{\Delta=0}$ but the proof is non trivial. Thus we obtain from (59) that the RS potential evaluated at its minimum $\widetilde{E}$ is a lower bound to the true asymptotic MI when $\Delta\!<\!\Delta_{\rm AMP}$,

[TABLE]

Combined with Theorem III.1, this yields Theorem III.2 for all $\Delta\!\in\![0,\Delta_{\rm AMP}]$.

IV-C3 The hard phase $\Delta\in[\Delta_{\rm AMP},\Delta_{\rm RS}]$

Notice first that $\Delta_{\rm AMP}\!\leq\!\Delta_{\rm Opt}$. Indeed since $\Delta_{\rm RS}\!\geq\!\Delta_{\rm AMP}$ (by their definitions) and both functions $i_{\rm RS}(E;\Delta)$ and $\lim_{L\to\infty}i^{\rm cs}$ are equal up to $\Delta_{\rm AMP}$, knowing that $i_{\rm RS}(E;\Delta)$ is analytic until $\Delta_{\rm RS}$ implies directly $\Delta_{\rm AMP}\!\leq\!\Delta_{\rm Opt}$.

Assume for a moment that $\Delta_{\rm Opt}\!=\!\Delta_{\rm RS}$. Thus both $\lim_{L\to\infty}i^{\rm cs}$ and $i^{\rm RS}(\widetilde{E};\Delta)$ are analytic on $]0,\Delta_{\rm Opt}[$ which, since they are equal on $[0,\Delta_{\rm AMP}]\!\subset\![0,\Delta_{\rm RS}]$, implies (by unicity of the analytic continuation) that they must be equal for all $\Delta\!<\!\Delta_{\rm RS}$. Concavity in $\Delta$ implies continuity of $\lim_{L\to\infty}i^{\rm cs}$ which allows to conclude that Theorem III.2 holds at $\Delta_{\rm RS}$ too.

IV-C4 Above the static phase transition $\Delta\geq\Delta_{\rm RS}$

For this noise regime, we have again $E^{(\infty)}\!=\!\widetilde{E}$ the global minimum of $i^{\rm RS}$. We can start again from (54) with $E^{(\infty)}$ replaced by $\widetilde{E}$ and apply a similar integration argument with the integral now running from $\Delta_{\rm RS}$ to $\Delta$ from which we get, after taking the limit $h\to 0$ (thanks to remark IV.2)

[TABLE]

The validity of the replica formula at $\Delta_{\rm RS}$ (just proved above under the assumption $\Delta_{\rm Opt}\!=\!\Delta_{\rm RS}$) is crucial to complete this argument. It allows to cancel $i^{\rm RS}(\widetilde{E};\Delta_{\rm RS})$ and $\lim_{L\to\infty}i^{\rm cs}|_{\Delta_{\rm RS}}$ which implies the inequality (60) for $\Delta\geq\Delta_{\rm RS}$. In view of Theorem III.1 this completes the proof of Theorem III.2.

It remains to show $\Delta_{\rm Opt}\!=\!\Delta_{\rm RS}$. This is where spatial coupling and threshold saturation come as new crucial ingredients.

IV-D Proof of $\Delta_{\rm Opt}\!=\!\Delta_{\rm RS}$ using spatial coupling

IV-D1 Spatial coupling

In order to show the equality of the thresholds, we need the introduction of two closely related spatially coupled CS models. Their construction is described by Fig. 1 which shows two measurement matrices replacing the homogeneous one of the CS model (1). When a SC measurement matrix is used, it naturally induces a decomposition of the signal ${\textbf{s}}=(s_{i})_{i=1}^{N}$ into $\Gamma$ “blocks” $c=1,\dots,\Gamma$ (with $\Gamma$ odd). Block $c$ of the signal corresponds to its $N/\Gamma$ components on which act the matrix entries inside the $c$-th “block-column”. It also induces a block decomposition of the measurement vector ${\textbf{y}}=(y_{\mu})_{\mu=1}^{M}$ into $\Gamma$ blocks $r=1,\dots,\Gamma$. Block $r$ corresponds to its $M/\Gamma$ components obtained from the product between the measurement matrix and the signal where the former is restricted to its $r$-th “block-row”.

The matrix on the left of Fig. 1 corresponds to taking periodic boundary conditions. This is called the periodic SC system (or model). On the right of Fig. 1 the SC system is “open". This open system is also called seeded SC system because we assume that the signal components belonging to the boundary blocks $\in\!\mathcal{B}$ are known and fixed. The number of boundary blocks is of the order of the coupling window $w$. The same construction was used for spatially coupled sparse superposition codes [36, 58] and we refer to these papers for more details. Introducing this “information seed” by fixing the boundaries is essential when proving the threshold saturation phenomenon described in the next subsection. The stronger variance at the boundaries of the open model help this information seed trigger a “reconstruction wave” that propagates the reconstructed signal inwards. This phenomenon (intimately related to crystal growth by nucleation) is what allows SC to reach such good practical performance, namely reconstruction of the signal by AMP at low measurement rate $\alpha$.

Here, the seeded SC model is not introduced for practical purposes. Instead, we introduce it to prove properties for the non coupled original CS model. Indeed, we are able to show that the seeded and original CS models have identical mutual informations in the thermodynamic limit. Proving this fact directly for the seeded model is rather cumbersome, and this is why the periodic model is first introduced as an intermediate step in the proof. Moreover because of threshold saturation, the algorithmic transition blocking AMP “is removed” for seeded SC models, allowing us to probe the “hard phase” (which is not hard anymore for the seeded SC system) by analysing the AMP algorithm.

Like for the CS model with (43), the SC model is associated to an interpolated Hamiltonian. In order to prove necessary concentrations, we also associate to each block of the signal an independent AWGN side channel. The Hamiltonian of the perturbed interpolated SC model is thus

[TABLE]

where $\{h_{c}\}$ are (small) snr values per block, and $\bm{\phi}$ is the SC measurement random matrix (see Fig. 1 for the structure of the variance of the matrix elements). As before, $\mathring{{\textbf{y}}}=[{{\textbf{y}}},{\widetilde{{\textbf{y}}}},{\widehat{{\textbf{y}}}},\bm{\phi}]$. Taking $\Gamma\!=\!1$ in this model, one recovers the (homogeneous) perturbed interpolated model (43). With slight abuse of notation, we will denote by $\langle\!-\!\rangle_{t,h}$ the Gibbs averages associated to this Hamiltonian (i.e. w.r.t the posterior (42) but with the Hamiltonian (62) replacing (43)). This is the same notation as for the Gibbs averages corresponding to (43), however the difference will be clear from the context.

IV-D2 Threshold saturation

The performance of AMP for the CS model, when SC matrices are used, is tracked by an MSE profile ${\textbf{E}}^{(t)}$, a vector $\in\![0,v]^{\Gamma}$ whose components are local MSE’s describing the quality of the reconstructed signal. More precisely, define ${\textbf{s}}_{c}$ as the vector made of the $L/\Gamma$ sections belonging to the $c$-th block of the signal, ${\widehat{{\textbf{s}}}}_{c}\!\!\!\!~{}^{(t)}$ its AMP estimate at iteration $t$. Then the $r$-th component of the profile is

[TABLE]

Consider the seeded SC system. The MSE profile ${\textbf{E}}^{(t)}$ can be computed by SE. The precense of the seed is reflected by $E_{r}^{(t)}=0$ for all $t$ if $r\!\in\!\mathcal{B}$. Apart from the boundary blocks we have

[TABLE]

with initialization $E_{r}^{(0)}=v$ for all $r\!\notin\mathcal{B}$, as required by AMP. Denote ${\textbf{E}}^{(\infty)}$ the fixed point profile of this SE recursion. Furthermore, denote $E_{\rm good}(\Delta)$ the smallest solution of the fixed point equation associated to the uncoupled SE recursion (23), or equivalently the smallest value of $E$ corresponding to an extrememum of $i^{\rm RS}(E;\Delta)$ (see remark II.2).

Definition IV.7** (AMP algorithmic threshold of the seeded SC model)**

The AMP algorithmic threshold for the seeded SC model is

[TABLE]

The order of the limits is essential.

It is proved in [36] by three of us that when AMP is used for seeded SC systems, threshold saturation occurs.

Lemma IV.8** (Threshold saturation)**

The AMP algorithmic threshold of the seeded SC system saturates to the RS threshold $\Delta_{\rm RS}$, that is

[TABLE]

In fact we can prove the equality holds, but we shall not need it here.

IV-D3 Invariance of the optimal threshold

Call the MI per section for the periodic and seeded SC systems, respectively, $i^{\rm per}_{\Gamma,w}$ and $i_{\Gamma,w}^{\rm seed}$. Using an interpolation method we will show in sec. VII the following asymptotic equivalence property between the coupled and original CS models. This non-trivial key result says that despite the 1-dimensional “chain” structure introduced in coupled models, the MI is preserved.

Theorem IV.9** (Invariance of the MI)**

The following MI limits exist and are equal for any (odd) $\Gamma$ and $w\!\in\!\{0,\ldots,(\Gamma\!-\!1)/2\}$:

[TABLE]

Moreover for any fixed $w$, we also have

[TABLE]

This implies straightforwardly that the information theoretic (or optimal) threshold $\Delta_{\rm Opt}^{\rm c}$ of the seeded SC model, defined as the first non-analyticity point, as $\Delta$ increases, of its asymptotic MI (i.e., the l.h.s of (69)), is the same as the one of the original uncoupled CS model (i.e., the r.h.s of (69)):

[TABLE]

This equality means that the phase transition occurs at the same threshold for the seeded SC and uncoupled CS models. This will be essential later on.

IV-D4 The inequality chain

We claim the following:

[TABLE]

and therefore we obtain the desired result

[TABLE]

The first inequality is Lemma IV.8. The second inequality follows from sub-optimality of AMP for the seeded SC system. The equality is (70) which follows from Theorem IV.9. The last inequality requires a final argument that we now explain.

Recall that $\Delta_{\rm Opt}\!<\!\Delta_{\rm AMP}$ is not possible. Let us show that $\Delta_{\rm RS}\!\in\,]\Delta_{\rm AMP},\Delta_{\rm Opt}[$ is also impossible. Since $\Delta_{\rm RS}\geq\Delta_{\rm AMP}$ this will imply $\Delta_{\rm RS}\geq\Delta_{\rm Opt}$. We proceed by contradiction so we suppose this is true. Then each side of (25) is analytic on $]0,\Delta_{\rm RS}[$ and since they are equal for $]0,\Delta_{\rm AMP}[\subset]0,\Delta_{\rm RS}[$, they must be equal on the whole range $]0,\Delta_{\rm RS}[$ by unicity of analytic continuation and also at $\Delta_{\rm RS}$ by continuity. For $\Delta\!>\!\Delta_{\rm RS}$ the fixed point of SE is $E^{(\infty)}\!=\!\widetilde{E}$ the global minimum of $i^{\rm RS}(E;\Delta)$, hence, the integration argument can be used once more on an interval $[\Delta_{\rm RS},\Delta]$ which implies that (25) holds for all $\Delta$. But then $i^{\rm RS}(\widetilde{E};\Delta)$ is analytic at $\Delta_{\rm RS}\!\in\,]\Delta_{\rm AMP},\Delta_{\rm Opt}[$ which is a contradiction.

The proof of the main Theorem III.2 is now complete. The rest of the paper contains the proofs of the other theorems and of the various intermediate lemmas.

V Guerra’s interpolation method: Proof of Theorem III.1

The goal of this section is to prove Theorem III.1. The ideas are to a fair extent identical to [27]. In this section, the Gibbs averages are taken considering the perturbed interpolated model (43).

First note that the interpolation (43) has been designed specifically so that, using (45), $i_{0,0}=i(\widetilde{{\textbf{S}}};\widetilde{{\textbf{Y}}})$ given by (13) where $\widetilde{{\textbf{y}}}$ comes from the denoising model discussed above (13) with noise variance equal to $\Sigma(E;\Delta)^{2}$. It implies from (15) that

[TABLE]

By the fundamental theorem of calculus, we have

[TABLE]

Using (73) this is equivalent to

[TABLE]

We derive a useful expression for the remainder $R_{t,h}$ which shows that it is negative up to a negligible term. For this purpose, it is useful to re-write $\psi(E;\Delta)$ given by (12) in a more convenient form. Using (39) and (40) one gets

[TABLE]

Using these two identities we obtain

[TABLE]

Let us now deal with the derivative term in the remainder. Straightforward differentiation gives

[TABLE]

These two quantities can be simplified using Gaussian integration by parts. For example, integrating by parts w.r.t $Z_{\mu}$,

[TABLE]

which allows to simplify $\mathcal{A}$,

[TABLE]

For $\mathcal{B}$ we proceed similarly with an integration by parts w.r.t $\widetilde{Z}_{i}$, and find

[TABLE]

Now, recalling the definitions (50) and (52) of ${\rm ymmse}_{t,h}$ and $E_{t,h}$, using Lemma IV.6 and (40), we obtain that for a.e $h$,

[TABLE]

Finally, combining (80), (79) and (76) we get for a.e $h$

[TABLE]

Since $\gamma(t)$ is an increasing function we see that, quite remarquably, $R_{t,h}$ is negative up to a vanishing term for a.e $h$. Since the limit $\lim_{L\to\infty}i_{1,h}$ exists we obtain from (75) that for a.e $h$

[TABLE]

Note from (43) that $i_{0,h}$ is independent of $L$ for all $h$. Moreover $\lim_{L\to\infty}i_{t,h}$ is concave (see Lemma IV.1) and thus continuous in any compact set containing $h=0$. Therefore this inequality is in fact valid for all $h$ in a compact set containing $h=0$. Now let $h\to 0$ and since $\lim_{h\to 0}\lim_{L\to\infty}i_{t,h}=\lim_{L\to\infty}\lim_{h\to 0}i_{t,h}$ (recall remark IV.2) we get that for any trial $E\in[0,v]$,

[TABLE]

which is the statement of Theorem III.1 recalling that $i_{1,0}=i^{\rm cs}$.

VI Linking the measurement and standard MMSE: Proof of Lemma IV.6

In this section we prove Lemma VI.1 below, which is a generalisation of Lemma IV.6. We place ourselves in the general setting where a SC measurement matrix is used and consider the perturbed interpolated SC model (62).

Lemma VI.1** (General MMSE relation)**

Consider model (62). Take a set $\mathcal{S}_{r}\subset\{1,\dots,M\}$ of rows of the measurement matrix where all the rows belong to a common block-row with index $r$ and where $|\mathcal{S}_{r}|=(M/\Gamma)^{u}$ with $0<u\leq 1$. Let ${\rm ymmse}_{{\cal S}_{r},t,h}$ the measurement MMSE associated to $\mathcal{S}_{r}$:

[TABLE]

Then it verifies for a.e. $h$

[TABLE]

where $r^{w}=\{r-w,\cdots,r+w\}$ and

[TABLE]

is the MMSE associated to the $c$-th block of the signal for the perturbed interpolated SC model.

Remark VI.2** (Homogeneous case)**

Lemma IV.6 is a special case, recovered from this more general lemma obtained for the SC model, by taking the homogeneous measurement matrix case, that is $\Gamma\!=\!1$, $w\!=\!0$ and ${\cal S}_{r=1}=\{1,\ldots,M\}$ (with $u=1$).

Remark VI.3** (Asymptotics)**

It is useful to keep in mind that $M=\alpha N=\alpha BL$ so $(M/\Gamma)^{u}=\mathcal{O}(L^{u})$.

Remark VI.4** (Other models)**

The proof shows that such generalised MMSE relations hold as long as the Hamiltonians are constituted of terms corresponding to AWGN channels and are carefully designed such that the Nishimori identities hold.

Proof:

The Nishimori identity (227) in appendix B is valid for the perturbed interpolated model (62), so

[TABLE]

Thus the per-block measurement MMSE verifies

[TABLE]

The second equality is obtained using an integration by part w.r.t the noise similarly to the steps (82)–(83). The last equality is obtained using (94). We also notice that (95) is equivalent to

[TABLE]

Indeed the equality between the second and third forms of ${\rm ymmse}_{\mathcal{S}_{r},t,h}$ in (95) implies

[TABLE]

and the right hand side is equal to $-{\rm ymmse}_{{\cal S}_{r},t,h}$ by the last equality in (95). This proves (96). Define $U_{\mu_{r}}\!\vcentcolon=\!\sqrt{\gamma(t)}\,[\bm{\Phi}\bar{\textbf{X}}]_{\mu_{r}}\!-\!Z_{\mu_{r}}$. Recalling that all the rows in $\mathcal{S}_{r}$ belong to block-row $r$ and using an integration by parts of (96) w.r.t $\phi_{\mu i}\!\sim\!\mathcal{N}(0,J_{r,c_{i}}/L)$ ($c_{i}$ being the block-column to which index $i$ belongs) leads to

[TABLE]

where the second equality comes from the construction of J (see the caption of Fig. 1) and the third equality comes from the identity (236) proved in appendix B combined with the Nishimori identity (223). Replacing $U_{\mu_{r}}$ by its expression,

[TABLE]

where we have defined

[TABLE]

together with $\mathcal{E}_{c}\vcentcolon=(\Gamma/L)\sum_{i_{c}=1}^{N/\Gamma}X_{i_{c}}\bar{X}_{i_{c}}$. By the law of large numbers (\Gamma/M)^{u}\sum_{\mu_{r}\in{\cal S}_{r}}z_{\mu_{r}}^{2}=1+\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}_{L}(1) almost surely as $L\to\infty$ so that using the Nishimori identity $\mathbb{E}[\langle{\cal E}_{c}\rangle_{t,h}]=E_{c,t,h}$, we reach

[TABLE]

For the other term ${\cal Y}_{{\cal S}_{r},2}$ we will show below that for a.e $h$,

[TABLE]

From (102), (101) and (98) we get that for a.e $h$,

[TABLE]

which is equivalent to (92).

It remains to prove (102). We prove in sec. VIII that $\mathcal{E}_{c}$ satisfies a concentration property, namely Proposition VIII.1. This proposition implies that for a.e $h$444 This is actually the point where the perturbation of the interpolated model is really needed. Ideally we would like to obtain this result for $h=0$. However continuity of this quantity may fail if one happens to be at a phase transition point and it is therefore difficult get a control of the limit $h\to 0$.,

[TABLE]

Set $a=\mathcal{E}_{c}$ and $b=Z_{\mu_{r}}[\bm{\Phi}\bar{\textbf{X}}]_{\mu_{r}}$. By Cauchy-Schwarz

[TABLE]

Because of (104) the variance of $a$ appearing under the square-root is \mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}_{L}(1) for a.e $h$. We now show the second moment of $b$ is bounded so that the $\mathcal{O}(\cdot)$ in (105) tends to [math] for a.e $h$. From Cauchy-Schwarz

[TABLE]

Recall $\bar{\textbf{x}}={\textbf{x}}-{\textbf{s}}$. Expanding the fourth power in $[\bm{\Phi}\bar{\textbf{X}}]_{\mu_{r}}^{4}$ the only terms that appear are of the form

[TABLE]

for some finite $m,n\geq 0$ (and in this case $\leq 4$). To bound such terms we use again Cauchy-Schwarz once more and then the Nishimori identity (225) in appendix B to obtain

[TABLE]

Now since s has i.i.d bounded components, $\bm{\phi}$ as well, and moreover are independent, the central limit theorem implies that $[\bm{\phi}{\textbf{s}}]_{\mu_{r}}$ converges in distribution to a random Gaussian variable with finite variance. Thus the moments in (108) are bounded and thus $\mathbb{E}[\langle b^{2}\rangle_{t,h}]$ as well. With our choice of $a$ and $b$ and these observations (105) implies from (100) that for a.e $h$,

[TABLE]

using again $\mathbb{E}[\langle{\cal E}_{c}\rangle_{t,h}]=E_{c,t,h}$. Finally from (109) and (96) we recognize that this relation is nothing else than (102). ∎

VII Invariance of the mutual information: Proof of Theorem IV.9

The interpolation method originates in the work of Guerra and Toninelli [56, 26] on the Sherrington-Kirkpatrick spin glass model and that of [59] for spin systems on sparse graphs. There are by now many variants of these methods, see for example [60, 27, 61, 62, 63, 64, 65, 39, 66, 67]. In order to prove Theorem IV.9 we introduce a new type of interpolation that we call sub-extensive interpolation method. This method borrows ideas from the Guerra-Toninelli interpolation [56] for dense systems and from the combinatorial approach of [65] suitable for sparse systems. In the Guerra-Toninelli approach one interpolates from one system to another by a global smooth change of the interactions. In contrast, in the combinatorial approach, one interpolates in discrete steps by changing one interaction (or constraint) at a time. Here we combine these two ideas: We will smoothly modify one step at a time a large but sub-extensive number of constraints along the interpolation path.

VII-A Invariance of the mutual information between the CS and periodic SC models

In this subsection we prove the first equality of Theorem IV.9. For that purpose, we will compare three models: The decoupled $w\!=\!0$ model, the SC $0\!<\!w\!<\!(\Gamma\!-\!1)/2$ model and the homogeneous $w\!=\!(\Gamma\!-\!1)/2$ model. In all cases, a periodic matrix (Fig. 1, left) is considered.

VII-A1 The $\rho$-ensembles

Recall that the periodic SC matrices are decomposed in $\Gamma\times\Gamma$ blocks (see Fig. 1). Focus only on the block-row decomposition. Block-rows are indexed by $r\in\{1,\dots,\Gamma\}$ and there are $M/\Gamma$ rows in each block-row. We consider a “virtual” thinner decomposition into sub-block-rows: each of the $\Gamma$ block-rows is decomposed in $(M/\Gamma)^{1-u}$ sub-block-rows with $(M/\Gamma)^{u}$ rows in each sub-block-row (here $0<u\ll 1$). The total number of such sub-block-rows is thus $\tau\!\vcentcolon=\!\Gamma(M/\Gamma)^{1-u}$. The number of rows belonging to one sub-block-row scales sub-extensively like ${\cal O}(L^{u})$ and their total number $\tau$ scales like ${\cal O}(L^{1-u})$. Let $\rho\!\in\!\{0,\dots,\tau\}$ and define a periodic SC matrix $\bm{\phi}_{\rho}$ as follows: $\tau\!-\!\rho$ of its sub-block-rows have a coupling window $w_{0}$ and the remaining $\rho$ ones have a coupling $w_{\tau}$. This defines the $\rho$-ensemble of measurement matrices (here the ordering of the sub-block-rows is irrelevant because a permutation amounts to order measurements differently and does not affect the MI).

For $w_{0}=0$ and $w_{\tau}=w$ the $\rho\in\{0,\dots,\tau\}$ ensembles interpolate between the decoupled $w=0$ ensemble corresponding to $\rho=0$ and the SC ensemble with coupling window $w$ corresponding to $\rho=\tau$. Similarly, for $w_{0}=w$ and $w_{\tau}=(\Gamma\!-\!1)/2$ the $\rho\in\{0,\dots,\tau\}$ ensembles interpolate between the SC ensemble with window $w$ corresponding to $\rho=0$ and the homogeneous ensemble with $w\!=\!(\Gamma\!-\!1)/2$ corresponding to $\rho=\tau$.

VII-A2 An intermediate basis ensemble

Our goal is to compare the MI of the $\rho$ and $(\rho\!+\!1)$-ensembles. We first construct a basis matrix $\bm{\phi}_{\rho}^{*}$ as follows. Consider $\bm{\phi}_{\rho}$ and select uniformly at random a block-row index $r\in\{0,\dots,\Gamma\}$. Inside this block-row, select uniformly at random a sub-block-row among the ones that have a coupling window $w_{0}$. Denote by $\mathcal{S}_{r}$ this sub-block-row. Then remove $\mathcal{S}_{r}$ from $\bm{\phi}_{\rho}$ (this process can be repeated until a sub-block-row with proper $w_{0}$ is found). This gives an intermediate basis matrix $\bm{\phi}_{\rho}^{*}$ with $\tau\!-\!\rho\!-\!1$ sub-block-rows with coupling $w_{0}$ and $\rho$ sub-block-rows with coupling $w_{\tau}$. Now, if we insert in place of $\mathcal{S}_{r}$ the sub-block-row with same index of a random SC matrix $\bm{\phi}_{0}$ with coupling window $w_{0}$, we get back a matrix from the $\rho$-ensemble. Instead, if we insert in place of $\mathcal{S}_{r}$ the sub-block-row of a random SC matrix $\bm{\phi}_{\tau}$ with coupling window $w_{\tau}$, we get a matrix from the $(\rho\!+\!1)$-ensemble. Matrices $\bm{\phi}_{0}$ and $\bm{\phi}_{\tau}$ will be denoted $\bm{\phi}_{q}$, $q\in\{0,\tau\}$.

VII-A3 Comparing $\rho$ and $(\rho\!+\!1)$-ensembles

We estimate the variation of MI when going from the basis system with matrix $\bm{\phi}_{\rho}^{*}$ to a system from the $\rho$-ensemble or $(\rho\!+\!1)$-ensemble. To do so we use a “smooth” and “global” interpolation. Define

[TABLE]

The last two terms are needed to use concentration properties similar to those of sec. VIII. The Hamiltonian is conditioned on the choice of the random block-row index $r$ and sub-block-row $\mathcal{S}_{r}$, and on all other usual quenched variables s, $\mathring{{\textbf{y}}}$ and $\bm{\phi}$. The interpolation parameters are $t\in[0,1]$ and $q\in\{0,\tau\}$. When $t\!=\!0$ there is no dependence on $q$ and this is the Hamiltonian of the basis system with matrix $\bm{\phi}^{*}_{\rho}$. Instead at $(t=1,q=0)$ this is the Hamiltonian of the $\rho$-ensemble, and at $(t=1,q=1)$ this is the Hamiltonian of the $(\rho\!+\!1)$-ensemble. Keep in mind that this Hamiltonian depends on $\{h_{c}\}$ but we leave this dependence implicit, and for the other quantities that we will introduce in this section as well, for the sake of readability.

Denote $i^{\rm per}_{q,t,r,\mathcal{S}_{r}}$ the MI associated to this general interpolated model when $r$ and $\mathcal{S}_{r}$ are fixed. For $t=0$ as noted above there is no dependence on $q$ so $i^{\rm per}_{q=0,t=0,r,\mathcal{S}_{r}}=i^{\rm per}_{q=\tau,t=0,r,\mathcal{S}_{r}}$ (and if we further average over $\mathcal{S}_{r}$ and $r$ we get the MI of the basis system). For $t=1$, if we average over $\mathcal{S}_{r}$ and $r$ we get $\mathbb{E}_{r,\mathcal{S}_{r}}[i^{\rm per}_{q=0,t=1,r,\mathcal{S}_{r}}]=i_{\rho}$ the MI of the $\rho$-ensemble and $\mathbb{E}_{r,\mathcal{S}_{r}}[i^{\rm per}_{q=\tau,t=1,r,\mathcal{S}_{r}}]=i_{\rho+1}$ the MI of the $(\rho\!+\!1)$-ensemble.

From the fundamental theorem of calculus

[TABLE]

so subtracting the $q=0$ and $q=\tau$ cases we obtain

[TABLE]

We now follow similar steps as in sec. V to compute the derivative w.r.t $t$. First one has

[TABLE]

where $\langle-\rangle_{q,t,r,\mathcal{S}_{r}}$ is the Gibbs average associated to the Hamiltonian (VII-A3). Define the measurement MMSE associated to the subset $\mathcal{S}_{r}$ and the normalized MMSE of a block $c$, as

[TABLE]

where recall $\{i_{c}\}$ are the components belonging to block $c$. Integrating by parts w.r.t the noise variables and using Lemma VI.1, a derivation similar to the one of equations (83) and (85), transforms (113) to

[TABLE]

for a.e. $h$, where recall $r^{w}\!\vcentcolon=\!\{r\!-\!w,\ldots,r\!+\!w\}$, $0<u\ll 1$, and \lim_{L\to\infty}\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}(L^{u-1})=0. We note that Lemma VI.1 applies here because Hamiltonian (VII-A3) is constructed so that all terms correspond to AWGN channels and Nishimori identities hold as well as the concentration property in Proposition VIII.1. Replacing (116) in (112) one gets for a.e. $h$

[TABLE]

The MMSE profile $E_{c,q,t,r,\mathcal{S}_{r}}$ depends only very weakly on $q$, $t$, $r$, $\mathcal{S}_{r}$ because $\mathcal{S}_{r}$ is sub-extensive, and this is actually the reason we chose it in such a way. More precisely, define E_{c,\rho}=(\Gamma/L)\mathbb{E}\big{[}\sum_{i_{c}=1}^{N/\Gamma}(\langle X_{i_{c}}\rangle_{\rho}-S_{i_{c}})^{2}\big{]} where $\langle\!-\!\rangle_{\rho}$ is the average corresponding to the Hamiltonian of the $\rho$-ensemble (obtained for $t=1$ and $q=0$). We prove in sec. VII-B that for a.e. $h$ (see Corollary VII.2),

[TABLE]

Since $M^{u}/(2\Gamma^{u}L)={\cal O}(L^{u-1})$, and performing explicitly the expectation $\mathbb{E}_{r,{\cal S}_{r}}$ (note that $E_{c,\rho}$ does not depend on ${\cal S}_{r}$) we then get for $u$ small enough and a.e $h$,

[TABLE]

Thanks to this identity we can easily compare the MI of the decoupled, coupled and homogeneous models.

VII-A4 Comparison of homogeneous and coupled models

We consider the $\rho\in\{0,\dots,\tau\}$ ensembles for the choice $w_{0}\!=\!w$ (with $0\!<\!w\!<\!(\Gamma-1)/2$) and $w_{\tau}\!=\!(\Gamma-1)/2$. With this choice and because of the periodicity of the model

[TABLE]

Therefore (119) and concavity of the logarithm immediately imply i_{\rho}\leq i_{\rho+1}+\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}(L^{u-1}). Now since $i_{\rho=0}=i_{\Gamma,w}^{\rm per}$, the MI of the periodic SC system with coupling $w$, and $i_{\rho=\tau}=i_{\Gamma,(\Gamma-1)/2}^{\rm per}$ we obtain i_{\Gamma,w}^{\rm per}\leq i_{\Gamma,(\Gamma-1)/2}^{\rm per}+\tau\,\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}(L^{u-1}) and since $\tau={\cal O}(L^{1-u})$ we get (for a.e. $h$)

[TABLE]

VII-A5 Comparison of decoupled and coupled models

We now consider the $\rho\in\{0,\dots,\tau\}$ ensembles for the choice $w_{0}\!=\!w$ (with $0\!<\!w\!<\!(\Gamma\!-\!1)/2$) and $w_{\tau}\!=\!0$. Because of the periodicity of the model and with this choice

[TABLE]

Convavity of the logarithm now implies i_{\rho}\geq i_{\rho+1}+\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}(L^{u-1}) for a.e $h$, which leads to

[TABLE]

VII-A6 Superadditivity and existence of thermodynamic limit

We note that the last inequality (123) is an instance of the superadditivity property of the mutual information. To understand this point consider the $L$ dependence of the mutual information $i^{\rm cs}_{L}$ of the basic system defined by the parameters $(L,N=LB,\alpha,B)$ (here $\alpha,B$ are held fixed). Consider three systems with sizes, $L$ even, $L_{1}=L/2$ and $L_{2}=L/2$. We interpret $L\,i^{\rm cs}_{L}$ as the mutual information of a coupled system with two blocks in the measurement matrix, i.e., $\Gamma=2$ and “complete coupling window” between these two blocks, and we interpret the sum $L_{1}\,i^{\rm cs}_{L_{1}}+L_{2}\,i^{\rm cs}_{L_{2}}$ as the total mutual information of two decoupled systems each of size $L/2=(N/\Gamma)B$. Therefore (123) is nothing else than the superadditivity statement i^{\rm cs}_{L}\geq\frac{L_{1}}{L}i^{\rm cs}_{L_{1}}+\frac{L_{2}}{L}i^{\rm cs}_{L_{2}}+\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}_{L}(1). By the same proof as above this inequality can easily be generalized to any three systems of sizes $L_{1}$, $L_{2}$, $L$ satisfying $L=L_{1}+L_{2}$. In fact with a bit more work (see Appendix F) we show that the terms \mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}_{L}(1) can be improved to $\mathcal{O}(L^{-\eta})$. Namely we have $i^{\rm cs}_{L}\geq\frac{L_{1}}{L}i^{\rm cs}_{L_{1}}+\frac{L_{2}}{L}i^{\rm cs}_{L_{2}}+\mathcal{O}(L^{-\eta})$ for some small $\eta>0$. Improved, but standard, versions of Fekete’s Lemma on super-additive sequences (see e.g., [68] or Appendix B of [65], and references therein for even stronger results) then imply the existence of the limit $\lim_{L\to+\infty}i^{\rm cs}(L)$.

We remark that in fact the inequalities (121) and (123) can also be improved to hold with an $\mathcal{O}(L^{-\eta})$ instead of \mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}_{L}(1). Indeed these results ultimately rest on Proposition VIII.1. However we do not directly need this improvement to conclude the proof in the next paragraph.

VII-A7 Combining everything and conclusion of proof

We now prove relation (68) in Theorem IV.9. First note that by construction the homogeneous model with $w=(\Gamma-1)/2$ is the model (1) and the decoupled model with $w=0$ is a union of $\Gamma$ independent models of size $(M/\Gamma)\times(N/\Gamma)$. Existence of the thermodynamic limit therefore implies

[TABLE]

On the other hand from (121) and (123) we have

[TABLE]

The existence of $\lim_{L\to+\infty}i^{\rm per}_{\Gamma,w}$ as well as relation (68) then follows from (124) and (125).

The careful reader will have noticed that a priori we proved this result for a.e. $h$. However by the usual concavity in $h$ arguments we know that these limits are uniform in $h$ and thus the limits $h\to 0$ and $L\to\infty$ can be exchanged. The result is thus valid for $h=0$. This concludes the proof of (68). We defer the proof of (69) in sec. VII-C.

VII-B Variation of MMSE profile

In this section we prove (118). This is done through a global interpolation.

Lemma VII.1** (MMSE variation 1)**

For a.e $h$ we have E_{c,q,t,r,\mathcal{S}_{r}}=E_{c,q,t=0,r,\mathcal{S}_{r}}+\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}_{L}(1) for all $c\in\{1,\ldots,\Gamma\}$. Observe from (VII-A3) that $E_{c,q,t=0,r,\mathcal{S}_{r}}$ is independent of $q\in\{0,\tau\}$.

Proof:

Note that thanks to the Nishimori identity (49)

[TABLE]

where we recall $\mathcal{E}_{c}=(\Gamma/L)\sum_{i_{c}=1}^{N/\Gamma}x_{i_{c}}(x_{i_{c}}-s_{i_{c}})$. Thus from the fundamental theorem of calculus

[TABLE]

where $G_{\nu}^{(q)}\vcentcolon=[\bm{\phi}_{q}\bar{\textbf{x}}]_{\nu}^{2}-[\bm{\phi}_{q}\bar{\textbf{x}}]_{\nu}z_{\nu}\sqrt{\Delta/s}$ and we used the Fubini theorem to exchange the order of the integrals (the integrand can be shown to be bounded and the integral is over a finite interval). Concentration of $\mathcal{E}_{c}$ as in Proposition VIII.1 is valid for the present model. Indeed all is needed in the proofs of sec. VIII are that all terms in the Hamiltonian can be interpreted as AWGN channels and Nishimori identities. Using that the $h$-integral of $\mathcal{E}_{c}$ concentrates we can check the integrand is \mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}_{L}(1) for a.e $h$ by arguments already used at the end of sec. VI. We give the main steps here for completeness.

Set $\delta\mathcal{E}_{c}\!\vcentcolon=\!\mathcal{E}_{c}\!-\!\mathbb{E}[\langle\mathcal{E}_{c}\rangle]$ where we drop the subscripts $q,s,r,\mathcal{S}_{r}$ in the Gibbs average. Using Cauchy-Schwarz

[TABLE]

We check that $\mathbb{E}[\langle G_{\nu}^{(q)})^{2}\rangle]$ is bounded. To do so we expand the square and use Gaussian integration by parts over the noise variables so that the only remaining terms are of the form $\mathbb{E}[[\bm{\Phi}_{q}{\textbf{S}}]_{\nu}^{n}\langle[\bm{\Phi}_{q}{\textbf{X}}]_{\mu}^{m}\rangle]$. Finally we proceed exactly as in (108) using Cauchy-Schwarz again and Nishimori identities to reduce such terms to estimates of quantities $\mathbb{E}[[\bm{\Phi}_{q}{\textbf{S}}]_{\lambda}^{\ell}]$ which are $\mathcal{O}(1)$. Now since by Proposition VIII.1 $\int_{\epsilon}^{a}dh\mathbb{E}[\langle\delta\mathcal{E}_{c}^{2}\rangle]\!=\!\mathcal{O}(L^{-1/10})$ for a.e $h$, we obtain that (128) is $\mathcal{O}(L^{-1/20})$. Using this and recalling that $|\mathcal{S}_{r}|=\mathcal{O}(L^{u})$, one reaches that (127) is ${\cal O}(L^{u-1/20})$. To conclude the proof of the lemma choose $u$ small enough and then use the same arguments (found below (138)) that led to the identity (104) (the MMSE’s are bounded uniformly in $L$). ∎

Corollary VII.2** (MMSE variation 2)**

For a.e $h$ we have E_{c,q,t,r,\mathcal{S}_{r}}=E_{c,\rho}+\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}_{L}(1) for all $c\in\{1,\ldots,\Gamma\}$.

Proof:

Recall that $E_{c,\rho}$ was defined as the MMSE of block $c$ for the $\rho$-ensemble, i.e. the Hamiltonian (VII-A3) corresponding to $t=1$ and $q=0$. Thus $E_{c,\rho}=E_{c,q=0,t=1,r,\mathcal{S}_{r}}$ and from Lemma VII.1 E_{c,q=0,t=1,r,\mathcal{S}_{r}}=E_{c,q=0,t,r,\mathcal{S}_{r}}+\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}_{L}(1). Thus E_{c,\rho}=E_{c,q=0,t,r,\mathcal{S}_{r}}+\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}_{L}(1) and this proves the corollary for $q=0$. For the case $q=\tau$ we note that Lemma VII.1 also implies E_{c,q=0,t=1,r,\mathcal{S}_{r}}=E_{c,q=0,t=0,r,\mathcal{S}_{r}}+\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}_{L}(1) and since the $t=0$ model is independent of $q$ we have $E_{c,q=0,t=0,r,\mathcal{S}_{r}}=E_{c,q=\tau,t=0,r,\mathcal{S}_{r}}$, so E_{c,\rho}=E_{c,q=\tau,t=0,r,\mathcal{S}_{r}}+\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}_{L}(1). Applying Lemma VII.1 once more we have E_{c,q=\tau,t=0,r,\mathcal{S}_{r}}=E_{c,q=\tau,t,r,\mathcal{S}_{r}}+\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}_{L}(1) and we conclude E_{c,\rho}=E_{c,q=\tau,t,r,\mathcal{S}_{r}}+\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}_{L}(1). ∎

VII-C Invariance of the mutual information between the periodic and seeded SC systems

We conclude this section by proving that in a proper limit, the seeded and periodic SC models have identical MI. We show that the difference between MI of these models is $\mathcal{O}(w/\Gamma)$ and thus vanishes when $\Gamma\!\to\!\infty$ for a fixed coupling window. As a consequence from (68) this proves equation (69) in the second part of Theorem IV.9.

The arguments below are essentially the same as those developed in [36, 58]. The only difference between the periodic and seeded SC systems is the boundary condition: The signal components belonging to the $8w$ boundary blocks $\in\mathcal{B}\!=\!\{1:4w\}\cup\{\Gamma-4w+1:\Gamma\}$ are known for the seeded system. Thus the Hamiltonians of the periodic and seeded SC systems, $\mathcal{H}^{\rm per}$ and $\mathcal{H}^{\rm seed}$, satisfy the identity

[TABLE]

recalling that $\{\mu_{r}\}$ is the set of “measurement indices” belonging to the block-row $r$ in the block decomposition of Fig. 1. The dependence of the Hamiltonians w.r.t the quenched random variables is implicit. Let $\mathcal{Z}^{\rm seed}$ and $\langle\!-\!\rangle_{\rm seed}$ be the partition function and posterior mean, respectively, associated to $\mathcal{H}^{\rm seed}({\textbf{x}})$:

[TABLE]

and similarly with $\mathcal{Z}^{\rm per}$, $\langle-\rangle_{\rm per}$ for $\mathcal{H}^{\rm per}({\textbf{x}})$. One obtains from (129) the following identities

[TABLE]

Using the convexity of the exponential, we get

[TABLE]

Due to the knowledge of the signal components at the $8w$ boundary blocks for the seeded system, one gets straightforwardly (set $\bar{{\textbf{x}}}=0$ in (130))

[TABLE]

Let us now study the lower bound. Using a Gaussian integration by part (as done in (82)) and the Nishimori identity (94), we obtain

[TABLE]

Therefore, both bounds tighten as $\Gamma\to\infty$ for any fixed $w$. Taking $L\!\to\!\infty$ and then $\Gamma\!\to\!\infty$ we get from (134), (135), (136) that

[TABLE]

Combining this with (68) yields (69) and proves the second part of Theorem IV.9.

VIII Concentration properties

The main goal of this section is to prove (104) holds. Recall $\mathcal{E}_{c}=(\Gamma/L)\sum_{i_{c}=1}^{N/\Gamma}x_{i_{c}}(x_{i_{c}}-s_{i_{c}})$.

Proposition VIII.1** (Concentration of ${\cal E}_{c}$)**

For any fixed $a>\epsilon>0$ the following holds:

[TABLE]

First let us clarify why this implies (104). The integrand in (138) is bounded uniformly in $L$ (in our setting this is obvious because the prior has bounded support). Therefore by Lebesgue’s dominated convergence theorem $\lim_{L\to\infty}\int_{\epsilon}^{a}dh\mathbb{E}[\langle(\mathcal{E}_{c}-\mathbb{E}[\langle\mathcal{E}_{c}\rangle_{t,h}])^{2}\rangle_{t,h}]=\int_{\epsilon}^{a}dh\lim_{L\to\infty}\mathbb{E}[\langle(\mathcal{E}_{c}-\mathbb{E}[\langle\mathcal{E}_{c}\rangle_{t,h}])^{2}\rangle_{t,h}]=0$ which implies that (104) holds for a.e. $h\in[\epsilon,a]$. Here $\epsilon>0$ is as small as we wish and $a$ can be arbitrarily large, thus for our purposes we can assert that (104) holds for a.e. $h>0$.

In order to make the proof more pedagogic, we will restrict to the case $\Gamma\!=\!1$, that is for model (43). All steps straightforwardly generalize to the SC case $\Gamma\!>\!1$, model (62), at the expense of more heavy notations. The proof relies on concentration properties of $\mathcal{L}$ given by (46). Once these are established, the concentration of

[TABLE]

follows as a consequence.

VIII-A Concentration of $\mathcal{L}$ on $\langle\mathcal{L}\rangle_{t,h}$

We first show the following lemma which expresses concentration of $\mathcal{L}$ around its posterior mean.

Lemma VIII.2** (Concentration of $\mathcal{L}$)**

For any fixed $a>\epsilon>0$, the following holds:

[TABLE]

Proof:

Let us evaluate the integral

[TABLE]

where the first equality follows from (48). The ${\cal O}(L^{-1})$ term comes from the second term in (48) which is ${\cal O}(1)$ (this is easily shown by integrating by parts the noise, and in addition the term with absolute value gives also a finite contribution). The proof is finished by noticing that the first derivatives of the MI are $\mathcal{O}(1)$ uniformly in $L$. This last point follows from general arguments on the $L\to\infty$ limit of concave in $h>0$ functions (recall Lemma IV.1). But an explicit check is also possible: from (47), (46) one obtains by integration by part w.r.t the standard Gaussian variable $\widehat{z}_{i}$

[TABLE]

where the second equality is due to the Nishimori identities, and the last uses that the signal is bounded. Finally, we obtain (140) from (141), (142) and Cauchy-Schwarz. ∎

VIII-B Concentration of $\langle\mathcal{L}\rangle_{t,h}$ on $\mathbb{E}[\langle\mathcal{L}\rangle_{t,h}]$

We will use a concentration statement for the free energy at fixed measurement realization. Recall that by definition $f_{t,h}(\mathring{{\textbf{y}}})\vcentcolon=-\ln(\mathcal{Z}_{t,h}(\mathring{{\textbf{y}}}))/L$ with $\mathring{{\textbf{y}}}$ the concatenation of all measurements and measurement matrix.

Proposition VIII.3** (Concentration of the free energy)**

For any $0<\eta<1/4$ we have

[TABLE]

A similar result has already been obtained in [27] for the CDMA problem. Here the proof has to be slightly generalised and uses the Ledoux-Talagrand and McDiarmid concentration theorems in conjunction (see appendix E).

Let us use the shorthand notations $f_{h}\vcentcolon=f_{t,h}(\mathring{{\textbf{y}}})$ and $\bar{f}_{h}\vcentcolon=\mathbb{E}[f_{t,h}(\mathring{{\textbf{Y}}})]$ which emphasize that we will look at small $h$-perturbations. By (47)

[TABLE]

The concavity of the free energy in $h$ (Lemma IV.1) allows to write the following inequalities for any $\delta>0$:

[TABLE]

Note that the difference between the derivatives appearing here cannot be considered small because at a first order transition point the derivatives have jump discontinuities. We now have all the necessary tools to show the second concentration.

Lemma VIII.4** (Concentration of $\langle\mathcal{L}\rangle_{t,h}$)**

For any fixed $a>\epsilon>0$ the following holds for any $0<\eta<1/4$:

[TABLE]

Proof:

Note that due to the concavity Lemma IV.1, we have

[TABLE]

Using (144), (146), (147), (149) we can write

[TABLE]

which implies

[TABLE]

We will now average over all quenched random variables and use Corollary VIII.3. We remark that by the central limit theorem combined with Cauchy-Schwarz, and as the noise and signal both have i.i.d components with finite first and second moments, $\mathbb{E}[|a_{1,2}|]=\mathcal{O}(L^{-1/2})$. Therefore

[TABLE]

Then integrating (152) and using (149) we get

[TABLE]

By the mean value theorem $\bar{f}_{a-\delta}-\bar{f}_{a}=-\delta\frac{d\bar{f}_{h}}{dh}|_{\widetilde{h}}$ for a suitable $\widetilde{h}\in[a-\delta,a]$. Since the first derivative of the free energy is $\mathcal{O}(1)$ (the argument is the same as for the MI, use (47) or (142)) we have $\bar{f}_{a-\delta}-\bar{f}_{a}=\delta\,\mathcal{O}(1)$. We proceed similarly for the other average free energy differences. Now choose $\delta=L^{-\eta/2}$. All this implies with (153) that

[TABLE]

Since $0<\eta<1/4$ the leading term is $\mathcal{O}(L^{-\eta/2})$ and this gives the result (148). ∎

VIII-C *Concentration of ${\cal E}$ on $\mathbb{E}[\langle\mathcal{E}\rangle_{t,h}]$:

Proof of Proposition VIII.1*

It will be convenient to use the following overlap $q_{{\textbf{x}},{\textbf{x}}^{\prime}}\vcentcolon=(1/L)\sum_{i=1}^{N}x_{i}x_{i}^{\prime}$. From (139), ${\cal E}=q_{{\textbf{x}},{\textbf{x}}}-q_{{\textbf{x}},{\textbf{s}}}$. We have that for any function $g$ such that $|g(x)|\leq 1$,

[TABLE]

where the last equality is obtained combining the triangle inequality with Lemma VIII.2 and Lemma VIII.4. Now consider $g={\cal E}/{\cal E}_{\rm max}$ where ${\cal E}_{\rm max}=2Bs_{\rm max}^{2}$ for a discrete prior. At the end of the section we show that

[TABLE]

Let us show that $|\mathcal{T}_{1}|$ is small. To do so we first notice the following property.

Lemma VIII.5** (Concentration of self-overlap)**

The self-overlap concentrates, i.e.,

[TABLE]

Proof:

By the Nishimori identity (225) we have $\mathbb{E}[\langle q_{{\textbf{x}},{\textbf{x}}}^{n}\rangle_{t,h}]=\mathbb{E}[q_{{\textbf{s}},{\textbf{s}}}^{n}]$. In particular $\mathbb{E}[\langle(q_{{\textbf{x}},{\textbf{x}}}-\mathbb{E}[\langle q_{{\textbf{x}},{\textbf{x}}}\rangle_{t,h}])^{2}\rangle_{t,h}]=\mathbb{E}[\langle q_{{\textbf{x}},{\textbf{x}}}^{2}\rangle_{t,h}]-\mathbb{E}[\langle q_{{\textbf{x}},{\textbf{x}}}\rangle_{t,h}]^{2}=\mathbb{E}[q_{{\textbf{s}},{\textbf{s}}}^{2}]-\mathbb{E}[q_{{\textbf{s}},{\textbf{s}}}]^{2}$. To conclude the proof, we use that the signal components are i.i.d with finite mean and variance, and thus the central limit theorem implies that $q_{{\textbf{s}},{\textbf{s}}}$ tends to a Gaussian random variable with finite mean and a variance $\mathbb{E}[q_{{\textbf{s}},{\textbf{s}}}^{2}]-\mathbb{E}[q_{{\textbf{s}},{\textbf{s}}}]^{2}={\cal O}(L^{-1})$. ∎

Lemma VIII.5 allows to prove that $|\mathcal{T}_{1}|=\mathcal{O}(L^{-1/2})$. From (157), using Cauchy-Schwarz and as $\mathbb{E}[\langle{\cal E}^{2}\rangle_{t,h}]={\cal O}(1)$ (the signal is bounded) we obtain

[TABLE]

Let us now consider $\mathcal{T}_{2}$ and show it is positive. From (158)

[TABLE]

where we used the Nishimori identity $\mathbb{E}[\langle q_{{\textbf{x}},{\textbf{x}}}q_{{\textbf{x}},{\textbf{x}}^{\prime}}\rangle_{t,h}]=\mathbb{E}[\langle q_{{\textbf{x}},{\textbf{x}}}q_{{\textbf{x}},{\textbf{s}}}\rangle_{t,h}]$. Now by convexity of a parabola,

[TABLE]

where in order to reach the last line we replaced the signal by an independent replica ${\textbf{x}}^{\prime\prime}$ thanks to the Nishimori identity and used that all replicas play the same role for the last equality. Using (156), (159), (161),

[TABLE]

Finally, combining this last inequality with (155) implies for fixed $a>\epsilon>0$,

[TABLE]

Now we choose the (sub-optimal) value $\eta=1/5$ (recall one must have $0<\eta<1/4$). This ends the proof of Proposition VIII.1.

It remains to prove (156)–(158). Consider $\mathbb{E}[\langle\mathcal{L}{\cal E}\rangle_{t,h}]-\mathbb{E}[\langle\mathcal{L}\rangle_{t,h}]\mathbb{E}[\langle\mathcal{E}\rangle_{t,h}]$. From (46) using integration by parts w.r.t $\widehat{z}_{i}$ one obtains for the first term

[TABLE]

Note here that as ${\cal E}$ is a function of x, s then $\mathbb{E}[\langle{\cal E}q_{{\textbf{x}},{\textbf{s}}}\rangle_{t,h}]\neq\mathbb{E}[\langle{\cal E}q_{{\textbf{x}},{\textbf{x}}^{\prime}}\rangle_{t,h}]$.

Now consider the second term. Using again an integration by part and the Nishimori identity $\mathbb{E}[\langle q_{{\textbf{x}},{\textbf{x}}^{\prime}}\rangle_{t,h}]=\mathbb{E}[\langle q_{{\textbf{x}},{\textbf{s}}}\rangle_{t,h}]$,

[TABLE]

The difference of (164) and (165) yields (156), (157), (158).

IX MMSE relation for the CS model: Proof of Theorem III.4

The goal of this section is to prove relation (26) in Theorem III.4. The general ideas that go in this proof are similar to those of sec. VI. There, we used a decoupling property based on the concentration of ${\cal E}_{c}$ for a.e. $h$ (see the proof of (102)). Here instead we will prove that

[TABLE]

concentrates for a.e. $\Delta$. The advantage of doing so is that we do not need to add an $h$-perturbation to the CS model and as a consequence we obtain (26) for a.e. $\Delta$ instead of (53) in Lemma IV.6. The reader may then wonder why we introduce the detour through the $h$-perturbation instead of directly showing (26) for a.e. $\Delta$. The reason is that in order to prove the invariance of the MI under spatial coupling we need the general relation (92) involving the measurement MMSE associated to a sub-extensive set $\mathcal{S}_{r}$ and the present proof does not go through (as such) for sub-extensive sets.

IX-A Concentration of $\mathcal{Q}$ on $\mathbb{E}[\langle Q\rangle]$

The concentration proof of ${\cal Q}$ closely follows the one of Proposition VIII.1. We first need to show the concentration of another intermediate quantity ${\cal M}$, analogously to the proof of the concentration of ${\cal E}$ which requires first to show the concentration of ${\cal L}$ (see sec. VIII). This quantity $\mathcal{M}$ naturally appears in the first and second derivatives of the MI w.r.t $\Delta$. We consider the CS model (1) with posterior given by (3). Posterior averages are as usual denoted by $\langle\!-\!\rangle$. A calculation of the first and second derivatives of the MI yields

[TABLE]

where

[TABLE]

Lemma IX.1** (Concentration of ${\cal M}$)**

For any fixed $a>\epsilon>0$ we have

[TABLE]

Proof:

The proof is similar to the one of Lemma VIII.2 so we only give a sketch. Integrating (168)

[TABLE]

The first derivatives of the MI is $\mathcal{O}(1)$ (uniformly in $L$). This is clear from the I-MMSE relation (9) as ${\rm ymmse}$ must be $\mathcal{O}(1)$. However we can also show it directly from the r.h.s of (167) by Nishimori’s identity and Cauchy-Schwarz. Cauchy-Schwarz is used for decoupling s and x when they appear together in averages, and the Nishimori identity is used for reducing all averages to moments of Gaussian random variables. Doing so, only terms of the form (108) appear which are all bounded independently of $L$. To show that the last term is $\mathcal{O}(1)$ we first integrate by parts $Z_{\mu}$ and then proceed through the Nishimori identity and the Cauchy-Schwarz inequality as just indicated (note that this term is identical to the last one entering in $\mathbb{E}[\langle{\cal M}\rangle]$). ∎

Now we prove the concentration of $\langle\mathcal{M}\rangle$ on $\mathbb{E}[\langle\mathcal{M}\rangle]$, namely

Lemma IX.2** (Concentration of $\langle\mathcal{M}\rangle$)**

For any $a>\epsilon>0$ we have

[TABLE]

Proof:

The proof is very similar to the arguments developed in sec. VIII-B. Consider the free energy $f^{\rm cs}({\textbf{y}},\bm{\phi})\vcentcolon=-\ln(\mathcal{Z}^{\rm cs}({\textbf{y}},\bm{\phi}))/L$. Note that Proposition VIII.3 apply since the CS model is a special case of the perturbed interpolated model where $h\!=\!0$ and $t\!=\!1$. These results are used below. We focus here on the rest of the proof which is analogous to that of Lemma VIII.4. The shorthand notations $f_{\Delta^{-1}}\vcentcolon=f^{\rm cs}({\textbf{y}},\bm{\phi})$ and $\bar{f}_{\Delta^{-1}}\vcentcolon=\mathbb{E}[f^{\rm cs}({\textbf{Y}},\bm{\Phi})]$ are convenient and emphasize that we are interested in small perturbations of $\Delta^{-1}$. We have

[TABLE]

By concavity of the MI (and thus of the free energy) in $\Delta^{-1}$ (see e.g. [49]),

[TABLE]

Set $C_{\Delta^{-1}}\vcentcolon=C_{\Delta^{-1}}^{+}+C_{\Delta^{-1}}^{-}\geq 0$. Using this remark with (174), by proceeding exactly as in (149)–(151) we obtain an inequality of the type (152). Taking the square and averaging we get

[TABLE]

where ${\cal K}\vcentcolon=\{\Delta^{-1}+\delta,\Delta^{-1},\Delta^{-1}-\delta\}$. We used the convexity of the squarre $(\sum_{i=1}^{p}v_{i})^{2}\leq p\sum_{i=1}^{p}v_{i}^{2}$ (here $p=5$) to get the second equality. Now note that $\mathbb{E}[a^{2}]={\cal O}(L^{-1})$. Indeed conditional on s, $[{\bm{\Phi}}{\textbf{s}}]_{\mu}$ and $[{\bm{\Phi}}{\textbf{s}}]_{\nu}$ are i.i.d Gaussian random variables with zero mean and finite variance, and by the central limit theorem $a$ tends in distribution to a zero mean Gaussian random variable with variance $\mathbb{E}_{{\textbf{Y}}|{\textbf{s}}}[a^{2}]={\cal O}(L^{-1})$, and thus after averaging over S, we have $\mathbb{E}[a^{2}]={\cal O}(L^{-1})$ too. Furthermore by the usual arguments $d\bar{f}_{\Delta^{-1}}/d\Delta^{-1}$ is $\mathcal{O}(1)$, and thus $C_{\Delta^{-1}}$ as well, uniformly in $L\to\infty$. Note also that $C_{\Delta^{-1}}\geq 0$. These remarks imply that $C_{\Delta^{-1}}^{2}=C_{\Delta^{-1}}\mathcal{O}(1)$. So the right hand side of (176) can be replaced by

[TABLE]

It remains to exploit the concentration of the free energy. From Proposition E.7 it easily follows that \mathbb{E}\big{[}\big{(}f_{\Delta^{-1}}-\bar{f}_{\Delta^{-1}}\big{)}^{2}\big{]}=\mathcal{O}(L^{-2\eta}) with $0<\eta<1/4$. So taking $\delta=L^{-\eta/2}$ we obtain for (177) $\mathcal{O}(L^{-\eta})+C_{\Delta^{-1}}\mathcal{O}(1)+\mathcal{O}(L^{-1})$. Integrating (176) for finite $a>\epsilon>0$, we get

[TABLE]

The integral equals differences of average free energies which, by the mean value theorem, are all $\mathcal{O}(\delta)=\mathcal{O}(L^{-\eta/2})$. Finally if we take $\eta=1/5$ this last estimate is ${\cal O}(L^{-1/10})$. This ends the proof of the lemma. ∎

We are now ready to show the concentration of $\mathcal{Q}$. The proof proceeds similarly as in sec. VIII-C.

Lemma IX.3** (Concentration of ${\cal Q}$)**

For any $a>\epsilon>0$ we have

[TABLE]

Proof:

Let $g$ be a function s.t $\mathbb{E}[\langle g^{2}\rangle]={\cal O}(1)$ that we choose later on. By Cauchy-Schwarz applied to $\int_{\epsilon}^{a}d\Delta^{-1}\mathbb{E}[\langle\!-\!\rangle]$,

[TABLE]

From Lemma IX.1, Lemma IX.2 and the triangle inequality (for the $L_{2}$-norm) this implies

[TABLE]

Now we apply this satement to $g=\cal Q$. A useful guide for the rest of the proof is to note the “symmetry” between $\cal Q$ in (166) and $\cal E$ in (139) and between $\cal L$ in (46) and $\cal M$ in (169) under the changes $x_{i}\leftrightarrow[{\bm{\phi}}{\textbf{x}}]_{\mu}$, $s_{i}\leftrightarrow[{\bm{\phi}}{\textbf{s}}]_{\mu}$, $h\leftrightarrow\Delta^{-1}$, $\widehat{z}_{i}\leftrightarrow z_{\mu}$. First, one checks $\mathbb{E}[\langle{\cal Q}^{2}\rangle]={\cal O}(1)$: This is shown by combining Cauchy-Schwarz with the Nishimori identity which leads to terms of the form (108), and from the discussion below (108) it follows that they are ${\cal O}(1)$. Second, we manipulate $\mathbb{E}[\langle\mathcal{M}g\rangle]-\mathbb{E}[\langle\mathcal{M}\rangle]\mathbb{E}[\langle g\rangle]$. Define the overlap $\widetilde{q}_{{\textbf{x}},{\textbf{x}}^{\prime}}\vcentcolon=(1/L)\sum_{\mu=1}^{M}[{\bm{\phi}}{\textbf{x}}]_{\mu}[{\bm{\phi}}{\textbf{x}}^{\prime}]_{\mu}$. Nishimori identities imply similar equalities as in (164), (165), and here we get

[TABLE]

where recall that ${\textbf{x}}^{\prime}$ is an independent replica. We now claim that $|\mathcal{T}_{1}^{\prime}|$ is small. Lemma VIII.5 extends to the self-overlap $\widetilde{q}_{{\textbf{x}},{\textbf{x}}}$ which thus concentrates. In addition, as already mentionned above $\mathbb{E}[\langle{\cal Q}^{2}\rangle]={\cal O}(1)$. Thus applying (159) with $q_{{\textbf{x}},{\textbf{x}}}$ replaced by $\widetilde{q}_{{\textbf{x}},{\textbf{x}}}$ and ${\cal E}$ replaced by ${\cal Q}$ we obtain $|\mathcal{T}_{1}^{\prime}|={\cal O}(L^{-1/2})$. We now consider $\mathcal{T}_{2}^{\prime}$. By the same steps used to obtain (160), (161) but applied to new overlaps we reach that $\mathcal{T}_{2}^{\prime}\geq 0$. From $|\mathcal{T}_{1}^{\prime}|={\cal O}(L^{-1/2})$ and $\mathcal{T}_{2}^{\prime}\geq 0$ together with (182) we have

[TABLE]

The result of the lemma then follows from (181) with $g={\cal Q}$ and (185). ∎

IX-B Proof of the MMSE relation (26)

We now have all the necessary tools for proving (26). We can specialise the proof in sec. VI for the non SC case $\Gamma=1$ and with $\mathcal{S}\to\{1,\ldots,M\}$, and $\gamma\to\Delta^{-1}$. The steps leading to (98), (99), (100) yield

[TABLE]

with

[TABLE]

where ${\cal E}$ is given by (139).

By the law of large numbers and the Nishimori identity to recognize the MSE through $\mathbb{E}[\langle{\cal E}\rangle]={\rm mmse}$, we get that

[TABLE]

We now turn to ${\cal Y}_{2}$. We will first re-express it in such a way that $\mathcal{Q}$ appears explicitly. An integration by parts w.r.t the noise yields (this is analogous to (82))

[TABLE]

Define $\bar{\textbf{x}}^{\prime}={\textbf{x}}^{\prime}-{\textbf{s}}$ where ${\textbf{x}}^{\prime}$ is an independent replica. Then

[TABLE]

where we recognized ${\cal Q}$ given by (166) to get the last line and claimed that the second term involving two replicas satisfies

[TABLE]

We defer the proof of (191) to the end of this section. Using for now (190) we have

[TABLE]

and by Cauchy-Swcharz, $\mathbb{E}[\langle{\cal E}^{2}\rangle]={\cal O}(1)$, and Lemma IX.3 we get that for a.e $\Delta$

[TABLE]

Recalling again that $\mathbb{E}[\langle{\cal E}\rangle]={\rm mmse}$ we obtain

[TABLE]

In addition expressing $\mathcal{Q}$ in terms of the overlap,

[TABLE]

The second equality uses the Nishimori identity $\mathbb{E}[\langle\widetilde{q}_{{\textbf{x}},{\textbf{x}}}\rangle]=\mathbb{E}[\widetilde{q}_{{\textbf{s}},{\textbf{s}}}]$, the third one $\mathbb{E}[\langle\widetilde{q}_{{\textbf{x}},{\textbf{x}}^{\prime}}\rangle]=\mathbb{E}[\langle\widetilde{q}_{{\textbf{x}},{\textbf{s}}}\rangle]$, and the last one follows from (8). From (194) and (195) we finally get for a.e. $\Delta$

[TABLE]

Now we can combine (186), (188), (196) to obtain for a.e $\Delta$

[TABLE]

This proves relation (26).

It remains to justify the claim (191). Recalling $q_{{\textbf{x}},{\textbf{x}}^{\prime}}\vcentcolon=(1/L)\sum_{i=1}^{N}x_{i}x_{i}^{\prime}$ we have from (139) that ${\cal E}=q_{{\textbf{x}},{\textbf{x}}}-q_{{\textbf{x}},{\textbf{s}}}$. Using $\widetilde{q}_{{\textbf{x}},{\textbf{x}}^{\prime}}\vcentcolon=(1/L)\sum_{\mu=1}^{M}[{\bm{\phi}}{\textbf{x}}]_{\mu}[{\bm{\phi}}{\textbf{x}}^{\prime}]_{\mu}$ as well we have

[TABLE]

We will show that the first difference on the r.h.s is $\mathcal{O}(L^{-1/2})$ and that the second vanishes. We start with the second difference. By the Nishimori identity, we can replace s by an independent replica ${\textbf{x}}^{\prime\prime}$ and get $\mathbb{E}[\langle q_{{\textbf{x}},{\textbf{s}}}\widetilde{q}_{{\textbf{s}},{\textbf{x}}^{\prime}}\rangle]-\mathbb{E}[\langle q_{{\textbf{x}},{\textbf{s}}}\widetilde{q}_{{\textbf{x}},{\textbf{x}}^{\prime}}\rangle]=\mathbb{E}[\langle q_{{\textbf{x}},{\textbf{x}}^{\prime\prime}}\widetilde{q}_{{\textbf{x}}^{\prime\prime},{\textbf{x}}^{\prime}}\rangle]-\mathbb{E}[\langle q_{{\textbf{x}},{\textbf{x}}^{\prime\prime}}\widetilde{q}_{{\textbf{x}},{\textbf{x}}^{\prime}}\rangle]$. But independent replicas are dummy variables in posterior averages $\langle\!-\!\rangle$ and can be interchanged ${\textbf{x}}\leftrightarrow{\textbf{x}}^{\prime\prime}$, so $\mathbb{E}[\langle q_{{\textbf{x}},{\textbf{x}}^{\prime\prime}}\widetilde{q}_{{\textbf{x}},{\textbf{x}}^{\prime}}\rangle]=\mathbb{E}[\langle q_{{\textbf{x}}^{\prime\prime},{\textbf{x}}}\widetilde{q}_{{\textbf{x}}^{\prime\prime},{\textbf{x}}^{\prime}}\rangle]$. Also $q_{{\textbf{x}}^{\prime\prime},{\textbf{x}}}=q_{{\textbf{x}},{\textbf{x}}^{\prime\prime}}$, thus the second difference in (198) vanishes. We now turn to the first difference in (198). By the Nishimori identity $\mathbb{E}[\langle\widetilde{q}_{{\textbf{x}},{\textbf{x}}^{\prime}}\rangle]=\mathbb{E}[\langle\widetilde{q}_{{\textbf{s}},{\textbf{x}}^{\prime}}\rangle]$, thus

[TABLE]

By the triangle inequality and Cauchy-Schwarz

[TABLE]

where in the last step we used the usual arguments combining the Nishimori identity, Cauchy-Schwarz and the discussion below (108) to assert that $\mathbb{E}[\langle\widetilde{q}_{{\textbf{x}},{\textbf{x}}^{\prime}}^{2}\rangle]=\mathbb{E}[\langle\widetilde{q}_{{\textbf{s}},{\textbf{x}}^{\prime}}^{2}\rangle]={\cal O}(1)$. Now we use the concentration of $q_{{\textbf{x}},{\textbf{x}}}$ in Lemma VIII.5 (that applies here as the Nishimori identiy is verified) to conclude from (200), $|\mathbb{E}[\langle q_{{\textbf{x}},{\textbf{x}}}\widetilde{q}_{{\textbf{x}},{\textbf{x}}^{\prime}}\rangle]\!-\!\mathbb{E}[\langle q_{{\textbf{x}},{\textbf{x}}}\widetilde{q}_{{\textbf{s}},{\textbf{x}}^{\prime}}\rangle]|={\cal O}(L^{-1/2})$. This ends the proof of claim (191).

X Optimality of AMP

At this stage, the main Theorem III.2 has been proven. In this section we show how to combine Corollary III.5 with the state evolution analysis of [23] for proving Theorem III.7. The proof is valid outside of the hard phase for $B=1$ and with a discrete prior s.t the RS potential $i^{\rm RS}(E;\Delta)$ in (15) has at most three stationary points. In order to prove this, we first need to show that the MSE of AMP verifies an identity of the form (26).

The results we state here are only rigorously valid for $B=1$ as the state evolution analysis of [23] is done in this case. Nevetheless, we conjecture that they are true for any finite $B$.

X-A MMSE relation for AMP

Recall the definition of the measurement and usual MSE of AMP given in (30) and (22). Their limits as $t\to\infty$ exist [23] and are denoted by $E^{(\infty)}$ and ${\rm ymse}_{\rm AMP}^{(\infty)}$.

Lemma X.1** (MMSE relation for AMP)**

We have almost surely (a.s)

[TABLE]

Proof:

Set

[TABLE]

This quantity appears in the Onsager term in (17). A general concentration result based on initial results in [23] and needed here is Equation (4.11) in [69]. This states that the following limit exists and is equal to

[TABLE]

a.s, where $\widetilde{\textbf{S}}\sim P_{0}$, $\widetilde{\textbf{Z}}\sim\mathcal{N}(0,\mathbf{I}_{B})$.

We start from (17) and replace the measurements by (1) (note the rescaling factor $\sqrt{\alpha B}$ between the CS model and the definition of AMP). Then we isolate the asymptotic measurement MSE of AMP and take the limit $t\to\infty$. We get

[TABLE]

a.s. Here $\langle{\textbf{x}},{\textbf{y}}\rangle=\sum_{i=1}^{M}x_{i}y_{i}$ and we used $\lim_{L\to\infty}\|{\textbf{z}}\|^{2}/M=1$ a.s by the central limit theorem. Recall the definition (22). We use Equation (3.19) in [23] which allows us to write that a.s,

[TABLE]

which implies

[TABLE]

Two other useful facts are Equation (2.1) in [23] and Lemma 4.1 of [69] which respectively become in the present case,

[TABLE]

a.s (we used (11) in the first equality). Using these two relations together with (206) allows to re-express (204) after simple algebra as

[TABLE]

We now claim that in the Bayesian optimal setting where the denoiser is defined by (19), we can express $w^{(\infty)}$ as a function of the MSE per section of AMP trough

[TABLE]

We will prove this formula in the next subsection. Plugging this formula in (209) directly implies (201). ∎

X-B Proof of the identity (210)

We compute explicitely (203) using (19). By computing the gradient we get

[TABLE]

where $\mathbb{E}$ is w.r.t $\widetilde{\textbf{S}}\sim P_{0}$, $\widetilde{{\textbf{Z}}}\sim\mathcal{N}(0,\mathbf{I}_{B})$ and where the posterior average $\langle\!-\!\rangle_{\tau_{t-1}}$ is

[TABLE]

For a discrete prior these integrals are in fact finite sums so, because of (207), it is clear that $\lim_{t\to\infty}\langle A({\textbf{X}})\rangle_{\tau_{t-1}}=\langle A({\textbf{X}})\rangle_{\Sigma(E^{(\infty)};\Delta)}$. With a bounded signal we can then easily apply the dominated convergence theorem to (211) to conclude

[TABLE]

The expected variance in this formula is nothing else than the MMSE of the effective AWGN channel ${\textbf{R}}=\widetilde{\textbf{S}}+\widetilde{\textbf{Z}}\Sigma(E^{(\infty)};\Delta)$. This can be seen explicitly by an application of the Nishimori identity (226). This leads with (203) and (207) to

[TABLE]

From (20), (23) we recognize that this MMSE actually corresponds to the fixed point of state evolution and is thus the asymptotic MSE of AMP $E^{(\infty)}$. Thus we find (210).

X-C Proof of Theorem III.7: Optimality of AMP

If $\Delta<\Delta_{\rm AMP}$ or $\Delta>\Delta_{\rm RS}$ we can assert, using the definitions of these thresholds and Remark II.2, that the MSE of AMP at its fixed point $t=\infty$ is also the global minimum of the potential $E^{(\infty)}=\widetilde{E}$ given by (16). Thus we replace $E^{(\infty)}$ by $\widetilde{E}$ in Lemma X.1. This combined with Corollary III.5 ends the proof of (36).

Then combining Theorem III.4 with (28) allows to identify (27), and thus to prove (35) as $E^{(\infty)}=\widetilde{E}$.

Appendix A I-MMSE formula

We give for completeness a short calculation to derive the I-MMSE formula (9) for our (structured) vector setting. Detailed proofs can be found in [43] and here we do not go through the technical justifications required to exchange integrals and differentiate under the integral sign.

Thanks to (II-A) the MI is represented as follows

[TABLE]

We exchange the y and s integrals. Then for fixed s we perform the change of variables ${\textbf{y}}\to{\textbf{z}}\sqrt{\Delta}+\bm{\Phi}{\textbf{s}}$. This yields

[TABLE]

where ${\textbf{Z}}\sim{\cal N}(0,\mathbf{I}_{M})$. We now perform the derivative $d/d\Delta^{-1}=(\sqrt{\Delta}/2)d/d\Delta^{-1/2}$. We use the statistical mechanical notation for the “posterior average”, namely for any quantity $A({\textbf{x}})$,

[TABLE]

Note that if $A$ does not depend on x then $\langle A\rangle=A$. We get

[TABLE]

Integrating the second term by parts w.r.t the Gaussian noise z we get that this term is equal to

[TABLE]

We therefore obtain

[TABLE]

In the last step we recognized that $\langle{\textbf{X}}\rangle$ is nothing else than the MMSE estimator $\mathbb{E}[{\textbf{X}}|\bm{\phi}{\textbf{s}}+{\textbf{z}}\sqrt{\Delta},\bm{\phi}]$ entering in the definition of the measurement MMSE (8).

Appendix B Nishimori identities

We collect here a certain number of Nishimori identities that are used throughout the paper. The basic identity from which other ones follow is (222) or equivalently (223) below. In fact this identity is just an expression of Bayes law.

B-1 Basic identity

Assume the vector s is distributed according to a prior $P_{0}({\textbf{s}})$ and its observation y is drawn from the conditional distribution $P({\textbf{y}}|{\textbf{s}})$. Furthermore, assume x is drawn from the posterior distribution $P({\textbf{x}}|{\textbf{y}})=P_{0}({\textbf{x}})P({\textbf{y}}|{\textbf{x}})/P({\textbf{y}})$. Then for any function $g({\textbf{x}},{\textbf{s}})$, using the Bayes formula,

[TABLE]

which is equivalent to

[TABLE]

where ${\textbf{X}},{\textbf{X}}^{\prime}$ are independent random vectors distributed according to the posterior distribution: we speak in this case about two “replicas”. Recalling that $\langle\!-\!\rangle$ is the posterior expectation $\mathbb{E}_{{\textbf{X}}|{\textbf{y}}}$ and $\mathbb{E}$ is the expectation w.r.t the quenched variables Y, S, relation (222) becomes

[TABLE]

where on the right hand side $\langle\!-\!\rangle$ stands for the average w.r.t the product distribution $P({\textbf{x}},{\textbf{x}}^{\prime}|{\textbf{y}})=P({\textbf{x}}|{\textbf{y}})P({\textbf{x}}^{\prime}|{\textbf{y}})$. We call this identity the Nishimori identity. Of course it remains valid for functions depending on more that one replica. In particular for $g({\textbf{x}},{\textbf{s}})=u({\textbf{x}})v({\textbf{s}})$ it implies

[TABLE]

Taking $u=1$ we have the very useful identity

[TABLE]

B-2 Second identity

We show that the expectation of the variance of the MMSE estimator equals the MMSE itself. Indeed,

[TABLE]

where we used $\mathbb{E}[\|{\textbf{S}}\|^{2}]=\mathbb{E}[\langle\|{\textbf{X}}\|^{2}\rangle]$ by (225) and $\mathbb{E}[{\textbf{S}}\cdot\langle{\textbf{X}}\rangle]=\mathbb{E}[\|\langle{\textbf{X}}\rangle\|^{2}]$ by (223).

B-3 Third identity

We now show

[TABLE]

The more elementary identity

[TABLE]

is derived similarly. The proof goes as follows. Consider the function

[TABLE]

and apply (223). We have $\mathbb{E}[\langle g_{1}({\textbf{X}},{\textbf{S}})\rangle]=\mathbb{E}[\langle g_{1}({\textbf{X}},{\textbf{X}}^{\prime})\rangle]$ with

[TABLE]

Now consider

[TABLE]

Applying (223) again we find

[TABLE]

From (230), (232) we have $\mathbb{E}[\langle g_{1}({\textbf{X}},{\textbf{S}})\rangle]=2\mathbb{E}[\langle g_{2}({\textbf{X}},{\textbf{S}})\rangle]$ which is (227).

B-4 Fourth identity

Consider a CS model with inverse noise variance $\gamma$. Recall $\bar{x}_{i}\vcentcolon=x_{i}-s_{i}$, $\bar{x}_{i}^{\prime}=x_{i}^{\prime}-s_{i}$ and set

[TABLE]

Note also that for a CS model with inverse noise variance $\gamma$, $z_{\mu}=\sqrt{\gamma}(y_{\mu}-[\bm{\phi}{\textbf{s}}]_{\mu})$. Then we have

[TABLE]

Indeed, from these previous relations

[TABLE]

using the Nishimori identity for the last step. From (234) one obtains the useful identity

[TABLE]

Appendix C Differentiation of $i^{\rm RS}$ with respect to $\Delta^{-1}$

In this section we prove (55). Recall that $\widetilde{E}(\Delta)$ is defined as the (global) minimiser of $i^{\rm RS}(E;\Delta)$ when it is unique. It is possible to show that in the first order phase transition scenario $\widetilde{E}(\Delta)$ is analytic except at $\Delta_{\rm RS}$555 This can be done through a direct application of the real analytic implicit function theorem to the function $f(E;\Delta)\vcentcolon=\partial i^{\rm RS}/\partial E$.. Therefore in particular $d\widetilde{E}/d\Delta$ is bounded for $\Delta\neq\Delta_{\rm RS}$. Thus since $\widetilde{E}(\Delta)$ is a solution of $\partial i^{\rm RS}(E;\Delta)/\partial E=0$, we have for $\Delta\neq\Delta_{\rm RS}$

[TABLE]

Now, from (15) we have

[TABLE]

where $i(\widetilde{\textbf{S}};\widetilde{\textbf{Y}})$ is given by (13). Using simple differentiation along with the chain rule, one gets

[TABLE]

Identity (55) follows directly from (238), (239) and (240).

We point out it is tedious to check the last equality in (240) by a direct computation of $\partial i(\widetilde{\textbf{S}};\widetilde{\textbf{Y}})/\partial\Sigma^{-2}$. But one can use the following trick. We know that $\widetilde{E}$ is a stationary point of $i^{\rm RS}(E;\Delta)$ so from (15) and (11),

[TABLE]

Then, by the chain rule one obtains

[TABLE]

Appendix D The zero noise limit of the mutual information

The replica formula for the MI is (15). We want to show that, for discrete bounded signals, $\lim_{\Delta\to 0}i^{\rm RS}(\widetilde{E};\Delta)=H(\widetilde{\textbf{S}})$ the Shannon entropy of $\widetilde{\textbf{S}}\sim P_{0}$, where recall that $\widetilde{E}$ is given by (16).

Assume first that $\widetilde{E}(\Delta)\to 0$ when $\Delta\to 0$. Then, according to (11), in this noiseless limit $\Sigma(\widetilde{E};\Delta)\to 0$. The zero noise limit of the denoising problem is then easily obtained. Indeed the explicit expression of (13) reads (where all vectors and norms are $B$-dimensional)

[TABLE]

The term $\sum_{\ell=1,\ell\neq k}^{K}$ is an average of terms smaller than $1$. Thus $\ln(p_{k}\exp(-\|{\textbf{z}}\|^{2}/2))\leq\ln(p_{k}\exp(-\|{\textbf{z}}\|^{2}/2)\!+\!1)$ and therefore the absolute value of the integrand is bounded above by

[TABLE]

This estimate is uniform in $\Sigma$ and integrable. Thus by Lebesgue’s dominated convergence theorem we can compute the limit of (243) by exchanging the limit and the integral. Obviously the $\Sigma\to 0$ limit of the term inside the logarithm is $p_{k}\exp(-\|{\textbf{z}}\|^{2}/2)$ and computing the resulting integral yields

[TABLE]

Assume further that $\widetilde{E}(\Delta)\to 0$ fast enough s.t $\lim_{\Delta\to 0}\widetilde{E}(\Delta)/\Delta=0$. Then from (12) we also have that $\lim_{\Delta\to 0}\psi(\widetilde{E};\Delta)=0$, and the desired result follows, namely that $\lim_{\Delta\to 0}i^{\rm RS}(\widetilde{E};\Delta)=H(\widetilde{\textbf{S}})$.

We thus only have to verify that $\lim_{\Delta\to 0}\widetilde{E}(\Delta)/\Delta=0$. Recall from Remark II.2 that $\widetilde{E}(\Delta)$ is a solution of

[TABLE]

We want to look at the behaviour of this fixed point equation when both $E\to 0$ and $\Delta\to 0$. When this is the case $\Sigma\to 0$. For a discrete prior ${\rm mmse}(\Sigma^{-2})={\cal O}(\exp(-c/\Sigma^{2}))$ where the constant $c>0$ is related to the minimum distance between alphabet elements (see next paragraph). Therefore for $E\to 0$, $\Delta\to 0$ and thus $\Sigma\to 0$, the solutions of (246) must satisfy $E/\Sigma^{2}\to 0$, in other words $E/(\Delta+E)\to 0$ or $(1+\Delta/E)^{-1}\to 0$. This can only happen if $E/\Delta\to 0$. Since for $\Delta<\Delta_{\rm AMP}$ there is a unique fixed point solution we deduce that necessarily $\lim_{\Delta\to 0}\widetilde{E}(\Delta)/\Delta=0$.

Note that for the above argument to hold {\rm mmse}(\Sigma^{-2})=\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}_{L}(\Sigma^{2}) is enough, and that this holds for discrete priors which have information dimension equal to zero [70]. Nevertheless we sketch here for completeness the proof that ${\rm mmse}(\Sigma^{-2})={\cal O}(\exp(-c/\Sigma^{2}))$ for a discrete prior. First we write down explicitly the ${\rm mmse}$:

[TABLE]

Let $\delta=\min_{j\neq j^{\prime}}\|{\textbf{a}}_{j}-{\textbf{a}}_{j^{\prime}}\|$ the minimum distance between any two distinct elements of the signal alphabet. We separate the integral in two sets: $\|{\textbf{z}}\|\leq\delta/(4\Sigma)$ and $\|{\textbf{z}}\|\geq\delta/(4\Sigma)$. To estimate the contribution of the second set we simply note that by the triangle inequality $\|\cdots\|^{2}\leq 4Bs_{\max}^{2}$ so the whole contribution is smaller than $4Bs_{\max}^{2}(2\pi)^{-B/2}\int_{\|{\textbf{z}}\|\geq\delta/(4\Sigma)}d{\textbf{z}}\exp(-\|{\textbf{z}}\|^{2}/2)={\cal O}(\exp(-\delta^{2}/(64\Sigma^{2}))$. For the first contribution we note that $\|{\textbf{z}}\|\leq\delta/(4\Sigma)$ implies $\|{\textbf{a}}_{\ell}-{\textbf{a}}_{k}\|^{2}-2\Sigma{\textbf{z}}\cdot({\textbf{a}}_{\ell}-{\textbf{a}}_{k})\geq\delta^{2}/2$ so that it is upper bounded by

[TABLE]

This concludes the argument.

In sec. IV-C we also use that $\lim_{\Delta\to 0}i^{\rm cs}=H(\widetilde{\textbf{S}})$. This follows from the fact that for discrete priors even a single noiseless measurement allows for a perfect reconstruction with high probability [70, 29].

Appendix E Concentration of the free energy: Proof of Proposition VIII.3

E-A Probabilistic tools

We have to prove concentration w.r.t the various types of quenched variables. We use two probabilistic tools in conjunction, namely the concentration inequality of Ledoux and Talagrand [71] for Gaussian random variables $\bm{\Phi},{\textbf{Z}},\widetilde{{\textbf{Z}}},\widehat{{\textbf{Z}}}$ and the McDiarmid inequality [72, 73] for the bounded random signal S. These inequalities are stated here for the convenience of the reader.

Proposition E.1** (Ledoux-Talagrand inequality)**

Let $f(U_{1},\dots,U_{P})$ a function of $P$ independent standardized Gaussian random variables which is Lipshitz w.r.t the Euclidean norm on the whole of $\mathbb{R}^{P}$, that is $|f(u_{1},\dots,u_{P})-f(u_{1}^{\prime},\dots,u_{P}^{\prime})|\leq K_{P}\|\textbf{u}-\textbf{u}^{\prime}\|$. Then

[TABLE]

Proposition E.2** (McDiarmid inequality)**

Let $f(U_{1},\cdots,U_{P})$ a function of $P$ i.i.d random variables that satisfies the following bounded difference property: For all $i=1,\ldots,P$, $|f(u_{1},\dots,u_{i},\ldots,u_{P})-f(u_{1},\dots,u_{i}^{\prime},\ldots,u_{P})|\leq c_{i}$ where $c_{i}>0$ is independent of $u_{1},\dots,u_{i},u_{i}^{\prime},\dots,u_{P}$. Then for any $r>0$,

[TABLE]

In our application the function (the free energy) is not Lipshitz over the whole of $\mathbb{R}^{P}$. To circumvent this technical problem we will use a result obtained in [27] (appendix I.A Theorem 9).

Proposition E.3** (Lipshitz extension)**

Let $f$ be a Lipshitz function over $G\subset\mathbb{R}^{P}$ with Lipshitz constant $K_{P}$. By the McShane and Whitney extension theorem [74] there exists an extension $g$ defined on the whole of $\mathbb{R}^{P}$ (so $g|_{G}=f$) which is Lipshitz with the same constant $K_{P}$ on the whole of $\mathbb{R}^{P}$.

Applying Proposition E.1 to $g$ yields the following (see [27] appendix I.A Lemma 7 for a detailed proof).

Proposition E.4** (Concentration of almost Lipshitz functions)**

Let $f$ be a Lipshitz function over $G\subset\mathbb{R}^{P}$ with Lipshitz constant $K_{P}$. Assume $0\in G$, $f(0)^{2}\leq C^{2}$, $\mathbb{E}[f^{2}]\leq C^{2}$ for some $C>0$. Then for $r\geq 6(C+\sqrt{PK_{P}})\sqrt{\mathbb{P}(G^{c})}$ we have

[TABLE]

E-B Concentration of the free energy with respect to the Gaussian quenched variables

We will apply these tools to show concentration properties of the free energy $f_{t,h}(\mathring{{\textbf{y}}})\vcentcolon=-\ln(\mathcal{Z}_{t,h}(\mathring{{\textbf{y}}}))/L$. The proof is decomposed in two parts. First we show thanks to the Ledoux-Talagrand inequality that for s fixed $f_{t,h}(\mathring{{\textbf{y}}})$ concentrates on $\mathbb{E}[f_{t,h}(\mathring{{\textbf{Y}}})|{\textbf{s}}]$ (so the expectation is over all Gaussian quenched variables $\bm{\Phi}$, Z, $\widetilde{\textbf{Z}}$, $\widehat{\textbf{Z}}$). Second, we show thanks to McDiarmid’s inequality that $\mathbb{E}[f_{t,h}(\mathring{{\textbf{Y}}})|{\textbf{s}}]$ concentrates on $\mathbb{E}[f_{t,h}(\mathring{{\textbf{Y}}})]$ (so the last expectation also includes the average over the signal distribution). Let us first fix s for all this sub-section.

Proposition E.5** (Concentration of the free energy w.r.t the Gaussian quenched variables)**

One can find two positive constants $c_{1}$ and $c_{2}$ (depending only on $s_{\rm max}$, $K$, $\Delta$, $\alpha$) s.t for any $r={\Omega}(e^{-c_{2}L^{1/2}})$, we have for any fixed s

[TABLE]

Proof:

The proof is the same as in [27] so we give the main steps only. Let

[TABLE]

where $D_{1}$ and $D_{2}$ will be chosen later as suitable powers of $N$ (x, s are always in the the discrete alphabet). We will see that the free energy is Lipshitz on ${\cal G}$, which will allow to use Proposition E.4. Let us first estimate $\mathbb{P}({\cal G}^{c})$, required in this Proposition. If $U$ is a zero mean Gaussian variable then $\mathbb{P}(|U|\geq\sqrt{A})\leq 4e^{-A/4}$. Therefore from the union bound

[TABLE]

Now conditional on x and s, $\sum_{i=1}^{N}\phi_{\mu i}(x_{i}-s_{i})$, $\mu=1,\dots,M$, are independent Gaussian random variables with zero mean and variance $a^{2}\!\leq\!4Bs_{\rm max}^{2}$. Let $X\!\sim\!{\cal N}(0,a^{2})$. The identity $\mathbb{E}[\exp(X^{2}\!/\!(16Bs_{\rm max}^{2})]\!=\!\sqrt{1\!+\!a^{2}\!/\!(8Bs_{\rm max}^{2}\!-\!a^{2})}$ $\leq\sqrt{2}$ thus implies

[TABLE]

Thus from Markov’s inequality for any given x, s in the discrete alphabet

[TABLE]

From the union bound we obtain (recall that here $K$ is the size of the discrete signal alphabet)

[TABLE]

Therefore from (254), (257) and the union bound we obtain

[TABLE]

This probability will be made small by a suitable choice of $D_{1}$ and $D_{2}$.

Now we must show that $f_{t,h}(\mathring{{\textbf{y}}})$ is Lipshitz on ${\cal G}$. To this end we set $\bm{\phi}^{0}=L^{1/2}\bm{\phi}$ in order to work only with standardized Gaussian random variables, and consider two sets of quenched variables $\bm{\phi}^{0}$, z, $\widehat{\textbf{z}}$, $\widetilde{\textbf{z}}$ and $\bm{\phi}^{{0}\prime}$, ${\textbf{z}}^{\prime}$, $\widehat{\textbf{z}}^{\prime}$, $\widetilde{\textbf{z}}^{\prime}$ belonging to the set ${\cal G}$. Proceeding exactly as in appendix I.E of [27], a slightly painful calculation leads to (recall the expression of the Hamiltonian (43))

[TABLE]

where $\|-\|_{F}$ is the Frobenius norm of the matrix and $\|-\|$ the Euclidean norm of the vectors. For the free energy difference we proceed as follows

[TABLE]

A similar argument yields a corresponding lower bound so that we get

[TABLE]

where for the last inequality we used $\sum_{i=1}^{n}\|{\textbf{x}}_{i}\|\leq\sqrt{n}\|({\textbf{x}}_{1},\ldots,{\textbf{x}}_{n})\|$ which follows from the convexity of the parabola ($({\textbf{x}}_{1},\ldots,{\textbf{x}}_{n})$ is the concatenation of $\{{\textbf{x}}_{n}\}$). Now we set $D_{1}=L^{\gamma}$ and $D_{2}$ a large enough constant. This gives a Lipshitz constant of order ${\cal O}(L^{-\frac{1-\gamma}{2}})$ for the Gaussian quenched variables in ${\cal G}$. It is perhaps worth to stress that the Lipshitz constant in the Ledoux-Talagrand inequality (and thus in Proposition E.4 as well) must be computed w.r.t the Euclidean norm and that the r.h.s of (261) is nothing else than the Euclidean norm of the $2N+M+NM=BL(2+\alpha+\alpha BL)={\cal O}(L^{2})$ components vector formed by all Gaussian quenched random variables.

Applying Proposition E.4 using (258) and $K_{P}={\cal O}(L^{-\frac{1-\gamma}{2}})$ we get for any $r={\Omega}(L^{(5+\gamma)/4}e^{-L^{\gamma}/8})={\Omega}(e^{-c_{2}L^{\gamma}})$ (for some small enough $c_{2}$) and fixed s,

[TABLE]

for some $c>0$. The choice $\gamma=1/2$ optimizes this estimate and yields (252) for a proper $c_{1}$.

We must finally check that $C$ in Proposition E.4 is ${\cal O}(1)$. Note that one can prove explicitly that $f_{t,h}(\mathring{{\textbf{y}}})|_{\bm{\Phi}={\textbf{z}}=\widetilde{\textbf{z}}=\widehat{\textbf{z}}=\bm{0}}^{2}$ and $\mathbb{E}[f_{t,h}(\mathring{{\textbf{y}}})^{2}|{\textbf{s}}]$ are finite. We sketch the argument which is essentially already found in [27]. For $\bm{\Phi}={\textbf{z}}=\widetilde{\textbf{z}}=\widehat{\textbf{z}}=\bm{0}$ we get a simple free energy on a discrete alphabet with no quenched variables. This is easily shown to be bounded. For the second quantity we proceed as follows. Since the Hamiltonian is positive we get the lower bound $f_{t,h}(\mathring{{\textbf{y}}})\geq-B\ln K$ (remember the free energy is defined with a minus in front of the logarithm and $K$ is the cardinality of the alphabet). On the other hand retaining only the term ${\textbf{x}}={\textbf{s}}$ in the statistical sum we obtain the upper bound $f_{t,h}(\mathring{{\textbf{y}}})\leq(\sum_{\mu}z_{\mu}^{2}+\sum_{i}\widetilde{z}_{i}^{2}+\widehat{z}_{i}^{2})/(2L)+\sqrt{h}s_{\rm max}(\sum_{i}|\widehat{z}_{i}|)/L$. Averaging over all Gaussian variables shows that $\mathbb{E}[f_{t,h}(\mathring{{\textbf{y}}})^{2}|{\textbf{s}}]$ must indeed be finite. ∎

E-C Concentration of the free energy with respect to the signal

We define the set of signal realizations

[TABLE]

where $0<\alpha<1$ is to be fixed later.

Proposition E.6** (Concentration of the free energy w.r.t the signal)**

One can find a constant $c_{3}>0$ (depending only on $s_{\rm max}$, $\alpha$ and $\Delta$) and $0<\alpha<1/2$ s.t for any ${\textbf{s}}\in{\cal S}_{\alpha}$,

[TABLE]

Proof:

Let ${\textbf{s}}\in{\cal S}_{\alpha}$. We show a bounded difference property for $\mathbb{E}[f_{t,h}(\mathring{{\textbf{Y}}})\mid{\textbf{s}}]/L$. Consider $s_{1},\dots,s_{i},s_{i}^{\prime},\dots,s_{N}$ in ${\cal S}_{\alpha}$ and estimate the corresponding Hamiltonian variation $\delta\mathcal{H}\!\vcentcolon=\!\mathcal{H}_{t,h}({\textbf{x}}|\mathring{{\textbf{y}}})\!-\!\mathcal{H}_{t,h}({\textbf{x}}|\mathring{{\textbf{y}}}^{\prime})$. From (43), $\delta\mathcal{H}\!=\!\delta\mathcal{H}_{\gamma}\!+\!\delta\mathcal{H}_{\lambda}\!+\!\delta\mathcal{H}_{h}$ where

[TABLE]

and $\delta\mathcal{H}_{h}$ is similar to $\delta\mathcal{H}_{\lambda}$ but with $h$ replacing $\lambda(t)$. These will be used a bit later, but first we need the following remark. Let $\mathcal{H}=\mathcal{H}_{t,h}({\textbf{x}}|\mathring{{\textbf{y}}})$, $\mathcal{H}^{\prime}=\mathcal{H}_{t,h}({\textbf{x}}|\mathring{{\textbf{y}}}^{\prime})$ and

[TABLE]

and similarly for $\langle A\rangle_{\mathcal{H}^{\prime}}$, $\mathcal{Z}^{\prime}$. We note

[TABLE]

so using the convexity of the exponential,

[TABLE]

Averaging over $\bm{\Phi}$, Z, $\widehat{\textbf{Z}}$, $\widetilde{\textbf{Z}}$ we obtain for fixed s and ${\textbf{s}}^{\prime}$ in $\cal{S}_{\alpha}$

[TABLE]

Recall $\delta\mathcal{H}=\delta\mathcal{H}_{\gamma}+\delta\mathcal{H}_{\lambda}+\delta\mathcal{H}_{h}$. Let us estimate $\mathbb{E}[\langle\delta\mathcal{H}_{\gamma}\rangle_{\mathcal{H},\mathcal{H}^{\prime}}\mid{\textbf{s}},{\textbf{s}}^{\prime}]$. From (265) we deduce

[TABLE]

In the last line we used boundedness of the signal. Similar equations lead to $|\mathbb{E}[\langle\delta\mathcal{H}_{\lambda}\rangle_{\mathcal{H},\mathcal{H}^{\prime}}\mid{\textbf{s}},{\textbf{s}}^{\prime}]|\!=\!|\mathbb{E}[\langle\delta\mathcal{H}_{h}\rangle_{\mathcal{H},\mathcal{H}^{\prime}}\mid{\textbf{s}},{\textbf{s}}^{\prime}]|\!=\!{\cal O}(1)$. Combining this with (271) and (272) we obtain the bounded difference property

[TABLE]

where we recall that ${\textbf{s}},{\textbf{s}}^{\prime}\in{\cal S}_{\alpha}$. The last step is a direct application of MacDiarmid’s inequality (Proposition E.2). ∎

E-D Proof of Proposition VIII.3

Proposition VIII.3 is a corollary of the following proposition.

Proposition E.7** (Concentration of the free energy)**

One can find a constant $c>0$ depending only on the parameters $s_{\rm max}$, $K$, $B$ and $\alpha$ s.t for any ${\textbf{s}}\in{\cal S}_{\alpha}$ and $0<\eta<1/4$,

[TABLE]

Proof:

Let ${\textbf{s}}\in{\cal S}_{\alpha}$. We have that the event $\{|f_{t,h}(\mathring{{\textbf{y}}})-\mathbb{E}_{{\textbf{S}}\in{\cal S}_{\alpha}}\mathbb{E}[f_{t,h}(\mathring{{\textbf{Y}}})\mid{\textbf{S}}]|\geq r\}$ implies by the triangle inequality the event $\{|f_{t,h}(\mathring{{\textbf{y}}})-\mathbb{E}[f_{t,h}(\mathring{{\textbf{Y}}})\mid{\textbf{s}}]|\geq r/2\}\cup\{|\mathbb{E}[f_{t,h}(\mathring{{\textbf{Y}}})\mid{\textbf{s}}]-\mathbb{E}_{{\textbf{S}}\in{\cal S}_{\alpha}}\mathbb{E}[f_{t,h}(\mathring{{\textbf{Y}}})|{\textbf{S}}]|\geq r/2\}$. Thus from the union bound and Propositions E.5 and E.6 we obtain

[TABLE]

We finally choose $r=L^{-\eta}$ and $\alpha=\eta$. Then the upper bound is $\mathcal{O}(e^{-cL^{1/2-2\eta}})$ for some $c>0$. This bound goes to [math] as $L$ for any $0<\eta<1/4$, which ends the proof. ∎

Now we can prove Proposition VIII.3. Let $b\vcentcolon=|f_{t,h}(\mathring{{\textbf{y}}})-\mathbb{E}_{{\textbf{S}}\in{\cal S}_{\alpha}}\mathbb{E}[f_{t,h}(\mathring{{\textbf{Y}}})\mid{\textbf{S}}]|$. Then using Proposition E.7 with $0<\eta<1/4$,

[TABLE]

where the last term follows by applying Cauchy-Schwarz and bounding $\mathbb{E}[b^{2}]$ by remarking that $b^{2}$ can be estimated by a polynomial-like function of Gaussian variables ${\textbf{z}},\widetilde{\textbf{z}},\widehat{\textbf{z}}$ (see the remarks at the end of the proof of Proposition E.5). Let us now estimate $\mathbb{P}({\cal S}_{\alpha}^{c})$ defined by (263). Applying successively Markov’s inequality, a convexity inequality and the Nishimori identity one obtains

[TABLE]

where the last equality is easy to check carefully using independence of random variables. Finally using the same bounds as above on $f_{t,h}(\mathring{{\textbf{y}}})$ we deduce that $\mathbb{E}_{{\textbf{S}}\in{\cal S}_{\alpha}}\mathbb{E}[b]=\mathbb{E}[|f_{t,h}(\mathring{{\textbf{y}}})-\mathbb{E}[f_{t,h}(\mathring{{\textbf{Y}}})]|]+{\cal O}(\mathbb{P}({\cal S}_{\alpha}^{c}))$ where the last average $\mathbb{E}$ includes all quenched variables, with S not anymore restricted to ${\cal S}_{\alpha}$. Then combining this with (276), (277) and recalling that $\alpha=\eta$ one obtains $\mathbb{E}[|f_{t,h}(\mathring{{\textbf{y}}})-\mathbb{E}[f_{t,h}(\mathring{{\textbf{Y}}})]|]={\cal O}(L^{-\eta})$ which is Proposition VIII.3.

Appendix F Super-additivity of the mutual information

As explained in the main text (Sec. VII-A, paragraph VII-A6) the same proof leading to (123) also yields the inequality i^{\rm cs}_{L}\geq\frac{L_{1}}{L}i^{\rm cs}_{L_{1}}+\frac{L_{2}}{L}i^{\rm cs}_{L_{2}}+\mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}_{L}(1). However to apply Fekete’s lemma (and its stronger versions) we need to improve the error term from \mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}_{L}(1) to $\mathcal{O}(L^{-\eta})$. In order to do so we note that an inspection of the arguments leading to (123) shows that they ultimately rest on proposition VIII.1. As a consequence it is easily seen that we can replace \mathchoice{{\scriptstyle\mathcal{O}}}{{\scriptstyle\mathcal{O}}}{{\scriptscriptstyle\mathcal{O}}}{\scalebox{0.7}{\scriptscriptstyle\mathcal{O}}}_{L}(1) by $\mathcal{O}(L^{-\eta})$ for some small $\eta>0$, if we look at the average of the mutual information over a small $h$-interval, $h\in[h_{0}-a_{L},h_{0}+a_{L}]$ where $h_{0}\in]0,1]$ and $a_{L}=\mathcal{O}(L^{-\eta/2})$ (in proposition VIII.1 the average is taken on a fixed interval but one can take an $L$ dependent interval at the expense of a worse $\eta>0$).

We obtain that for any $L_{1}$, $L_{2}$, $L$ such that $L=L_{1}+L_{2}$,

[TABLE]

By an explicit calculation of the $h$-derivative in equation (142) we know that $i_{L}^{\rm cs}(h)$ is $h$-Lipshitz with constant $\kappa$ independent of $L$. Therefore we have for the term on the l.h.s

[TABLE]

For the two terms on the r.h.s we have a similar statement. Combining with (278) we find

[TABLE]

which is the desired super-additivity.

Appendix G Unicity of $\Delta_{\rm RS}$ under the assumptions on $i_{\rm RS}(E;\Delta)$

First we argue that $\widetilde{E}(\Delta)$, and by way of consequence $i^{\rm RS}(\widetilde{E}(\Delta);\Delta)$, are analytic in $\Delta$ except at $\Delta_{\rm RS}$ where the ${\rm argmin}$ in (16) is not unique. For models with a discrete prior, (15) can be shown to be a real analytic function in each argument $E>0$, $\Delta>0$. Set $f(E,\Delta)=\partial_{E}i_{\rm RS}(E;\Delta)$ for the partial derivative w.r.t $E$. Consider a stationary point, i.e., a solution $E_{*}(\Delta)$ of the equation $f(E,\Delta)=0$. As long as $\partial_{E}f(E,\Delta)|_{E_{*}(\Delta)}\neq 0$ for $\Delta\in U\subset\mathbb{R}_{+}$ an open set, i.e., as long as this solution is not a horizontal inflexion point, the implicit function theorem for real analytic functions implies that $E_{*}(\Delta)$ is analytic w.r.t $\Delta\in U$. One concludes that the solutions of the stationary point equation are analytic branches over their domain of definition as long as they do not merge (because if a minimum and a maximum merge a horizontal inflexion point appears). This allows to conclude that $\widetilde{E}(\Delta)$ must be analytic in $\Delta$ except at $\Delta_{\rm RS}$ where the ${\rm argmin}\,i^{\rm RS}(E;\Delta)$ is not unique.

When there are at most three stationary points for (15) the non-unicity of the ${\rm argmin}$ occurs in two possible ways: (i) it continuously bifurcates into a maximum and minimum pair, or; (ii) it jumps from one analytic branch to the other. Possibility (i) corresponds to a continuous phase transition in the MMSE, and possibility (ii) corresponds to a first order phase transition. Let us now argue that $\Delta_{\rm RS}$ is unique.

We define the gap as the quantity: $+\infty$ when the ${\rm argmin}\,i^{\rm RS}(E;\Delta)$ is unique and there is no other local minimum; as $i^{\rm RS}(E_{\rm loc}(\Delta);\Delta)-i^{\rm RS}(\widetilde{E}(\Delta);\Delta)$ when the ${\rm argmin}\,i^{\rm RS}(E;\Delta)$ is unique and there is a local minimum $E_{\rm loc}(\Delta)\neq\widetilde{E}(\Delta)$; as [math] when the ${\rm argmin}\,i^{\rm RS}(E;\Delta)$ takes two values. Now, by explicit calculation we have for any stationary point $E_{*}(\Delta)$ an I-MMSE type of relation,

[TABLE]

This relation is derived by explicit calculation in Appendix C for the global minimum $\widetilde{E}(\Delta)$, but the calculation is exactly the same for all stationary points. Therefore the variation of the gap with $\Delta$ (when it is neither zero or infinite) is given by

[TABLE]

For small enough $\Delta$ one can argue that the gap is infinite. Suppose that a local minimum appears as we increase $\Delta$. First assume that $E_{\rm loc}(\Delta)\geq\widetilde{E}(\Delta)$. Then the last formula shows that the gap must diminish. Suppose it diminishes until a value of $\Delta_{\rm RS}$ where it vanishes. At this point the local and global minima switch roles and for slightly higher $\Delta$ we necessarily have $\widetilde{E}(\Delta)\geq E_{\rm loc}(\Delta)$ which implies that the gap must increase. A bit of thought shows that the gap necessarily will increase to infinity (either the local minimum continues to exist and by analyticity the gap cannot stay constant, or the local minimum merges with the local maximum and the gap becomes infinite by definition). Finally consider the situation where as $\Delta>0$ increases the local minimum appears s.t. $E_{\rm loc}(\Delta)\geq\widetilde{E}(\Delta)$. Then the gap must increase and arguing as before we can see that it will increase to infinity.

Acknowledgments

We thank Marc Lelarge and Andrea Montanari for clarifications on state evolution for the vectorial case $B\geq 2$ and Andrea Montanari for helpful discussions related to Section X. We also thank Léo Miolane for indicating how to extend our results to a wider class of prior distributions (see remark III.6). This work was done while Jean Barbier and Mohamad Dia were affiliated with EPFL. Jean Barbier and Mohamad Dia acknowledge funding from the Swiss National Science Foundation (grant 200021-156672). Florent Krzakala thanks the Simons Institute in Berkeley for its hospitality and acknowledges funding from the European Union (FP/2007-2013/ERC grant agreement 307087-SPARCS).