A Tight Degree 4 Sum-of-Squares Lower Bound for the Sherrington-Kirkpatrick Hamiltonian

TL;DR

This paper proves a tight degree 4 sum-of-squares lower bound for certifying upper bounds on the Sherrington-Kirkpatrick Hamiltonian, showing it cannot do better than the spectral maximum, and proposes a conjecture for higher degrees.

Contribution

It establishes a tight degree 4 SOS lower bound for the SK Hamiltonian and introduces a conjecture for extending lower bounds to higher degrees using pseudomoment constructions.

Findings

Degree 4 SOS cannot certify bounds below the spectral maximum.

With high probability, the lower bound matches the maximum eigenvalue asymptotically.

Proposes a conjecture for lower bounds at any fixed degree as N grows.

Abstract

We show that, if $\mathbf{W} \in \mathbb{R}^{N \times N}_{\mathsf{sym}}$ is drawn from the gaussian orthogonal ensemble, then with high probability the degree 4 sum-of-squares relaxation cannot certify an upper bound on the objective $N^{-1} \cdot \mathbf{x}^\top \mathbf{W} \mathbf{x}$ under the constraints $x_i^2 - 1 = 0$ (i.e. $\mathbf{x} \in \{ \pm 1 \}^N$) that is asymptotically smaller than $\lambda_{\max}(\mathbf{W}) \approx 2$. We also conjecture a proof technique for lower bounds against sum-of-squares relaxations of any degree held constant as $N \to \infty$, by proposing an approximate pseudomoment construction.

Equations

$$\mathsf{M}(\mathbf{W}):=\frac{1}{N}\underset{\mathbf{x}\in \{\pm 1{\}}^N}{\max }{\mathbf{x}}^{\top }\mathbf{W}\mathbf{x}.$$

$$\underset{N\to \infty }{\lim }\mathbb{E}\mathsf{M}(\mathbf{W})=:2{\mathsf{P}}_{\ast }\approx \mathrm{1.5264.}$$

$$\underset{N\to \infty }{\lim }\mathbb{P}\left[\frac{1}{N}{\mathbf{x}}^{\top }\mathbf{W}\mathbf{x}\ge 2{\mathsf{P}}_{\ast }-ϵ\right]=1.$$

$${\mathcal{C}}^N:=\mathsf{conv}\left(\left\{\mathbf{x}{\mathbf{x}}^{\top }:\mathbf{x}\in \{\pm 1{\}}^N\right\}\right),$$

$$\mathsf{M}(\mathbf{W})=\frac{1}{N}\underset{\mathbf{M}\in {\mathcal{C}}^N}{\max }⟨\mathbf{W},\mathbf{M}⟩.$$

$${\mathcal{C}}^N\subseteq \{\mathbf{M}\in {\mathbb{R}}_{\mathsf{sym}}^{N\times N}:\mathbf{M}⪰0,\mathrm{Tr}(\mathbf{M})=N\}=:{\mathcal{E}}_{\mathsf{spec}}^N.$$

$${\mathcal{E}}_2^N\supset {\mathcal{E}}_4^N\supset \cdots \supset {\mathcal{E}}_{N+\text{mathbbold}1\{N\text{ odd}\}}={\mathcal{C}}^N.$$

$$Z_{\{i\}\{j\}}=M_{ij}\text{ for all }i,j\in [N]$$

$${\mathsf{SOS}}_d(\mathbf{W}):=\frac{1}{N}\underset{\mathbf{M}\in {\mathcal{E}}_d^N}{\max }⟨\mathbf{W},\mathbf{M}⟩.$$

$$\underset{N\to \infty }{\lim }\mathbb{P}\left[{\mathsf{SOS}}_2(\mathbf{W})\ge 2-ϵ\right]=1.$$

$${\mathbf{M}}^{(\delta )}={\mathbf{M}}^{(\delta )}(\mathbf{W}):={\mathbf{D}}^{-1/2}\mathbf{P}{\mathbf{D}}^{-1/2}.$$

$$\underset{N\to \infty }{\lim }\mathbb{P}\left[\frac{1}{N}⟨{\mathbf{M}}^{(\delta )}(\mathbf{W}),\mathbf{W}⟩\ge 2-ϵ\right]=1.$$

$${\mathbf{M}}^{(\alpha ,\delta )}(\mathbf{W}):=(1-\alpha ){\mathbf{M}}^{(\delta )}(\mathbf{W})+\alpha {\mathbf{I}}_N.$$

$$\underset{N\to \infty }{\lim }\mathbb{P}\left[{\mathbf{M}}^{(\alpha ,\delta )}(\mathbf{W})\in {\mathcal{E}}_4^N\right]=1.$$

$$\underset{N\to \infty }{\lim }\mathbb{P}\left[{\mathsf{SOS}}_4(\mathbf{W})\ge 2-ϵ\right]=1.$$

$${\mathsf{SOS}}_4(\mathbf{W})\ge \frac{1}{N}⟨{\mathbf{M}}^{(\alpha ,\delta )}(\mathbf{W}),\mathbf{W}⟩\ge 2-\frac{2ϵ}{3}+\frac{1}{N}\alpha \mathrm{Tr}(\mathbf{W}).$$

$$\text{“}\mathbb{P}\left[\text{quantity}\le O_{\delta ,K}(N)\right]\ge 1-O_{\delta ,K}(N^{-K}).\text{”}$$

$$\mathsf{isovec}(\bm{A})\colonequals\left[\begin{array}[]{c}\mathsf{diag}(\bm{A})\\ \sqrt{2}\cdot\mathsf{offdiag}(\bm{A})\end{array}\right],$$

$$\mathbb{P}\left[\max_{i,j\in[N]}\left\{\begin{array}[]{lll}\left|P_{ii}-\delta\right|&\text{if}&i=j\\ \left|P_{ij}\right|&\text{if}&i\neq j\end{array}\right\}\leq O_{\delta,K}\left(\sqrt{\frac{\log N}{N}}\right)\right]\geq 1-O_{\delta,K}(N^{-K}).$$

$$P_{ii}\stackrel{(\mathrm{d})}{=}\frac{\Vert {\mathbf{g}}^{(r)}{\Vert }_2^2}{\Vert {\mathbf{g}}^{(r)}{\Vert }_2^2+\Vert {\mathbf{g}}^{(N-r)}{\Vert }_2^2},$$

$$(\mathbf{v},\mathbf{w})\stackrel{(\mathrm{d})}{=}\left(\frac{\mathbf{g}}{\Vert \mathbf{g}{\Vert }_2},\frac{\mathbf{h}-\frac{⟨\mathbf{g},\mathbf{h}⟩}{\Vert \mathbf{g}{\Vert }_2^2}\mathbf{g}}{{\Vert \mathbf{h}-\frac{⟨\mathbf{g},\mathbf{h}⟩}{\Vert \mathbf{g}{\Vert }_2^2}\mathbf{g}\Vert }_2}\right).$$

$$P_{ij}\stackrel{(\mathrm{d})}{=}\frac{⟨{\mathbf{g}}^{(r)},{\mathbf{h}}^{(r)}⟩-⟨\mathbf{g},\mathbf{h}⟩\frac{\Vert {\mathbf{g}}^{(r)}{\Vert }_2^2}{\Vert {\mathbf{g}}^{(r)}{\Vert }_2^2+\Vert {\mathbf{g}}^{(N-r)}{\Vert }_2^2}}{\Vert \mathbf{g}{\Vert }_2{\Vert \mathbf{h}-\frac{⟨\mathbf{g},\mathbf{h}⟩}{\Vert \mathbf{g}{\Vert }_2^2}\mathbf{g}\Vert }_2}.$$

$$\left|\frac{⟨{\mathbf{g}}^{(r)},{\mathbf{h}}^{(r)}⟩-⟨\mathbf{g},\mathbf{h}⟩\frac{\Vert {\mathbf{g}}^{(r)}{\Vert }_2^2}{\Vert {\mathbf{g}}^{(r)}{\Vert }_2^2+\Vert {\mathbf{g}}^{(N-r)}{\Vert }_2^2}}{\Vert \mathbf{g}{\Vert }_2{\Vert \mathbf{h}-\frac{⟨\mathbf{g},\mathbf{h}⟩}{\Vert \mathbf{g}{\Vert }_2^2}\mathbf{g}\Vert }_2}\right|\le \frac{|⟨{\mathbf{g}}^{(r)},{\mathbf{h}}^{(r)}⟩|+|⟨\mathbf{g},\mathbf{h}⟩|}{\Vert \mathbf{g}{\Vert }_2(\Vert \mathbf{h}{\Vert }_2-\frac{|⟨\mathbf{g},\mathbf{h}⟩|}{\Vert \mathbf{g}{\Vert }_2})}.$$

$$\mathbb{P}\left[\max_{i,j\in[N]}\left\{\begin{array}[]{lll}\left|M_{ii}-1\right|&\text{if}&i=j\\ \left|M_{ij}\right|&\text{if}&i\neq j\end{array}\right\}\leq O_{\delta,K}\left(\sqrt{\frac{\log N}{N}}\right)\right]\geq 1-O_{\delta,K}(N^{-K}).$$

$$\mathbb{P}\left[\Vert \mathbf{M}{\Vert }_{\mathsf{op}}\le {\delta }^{-1}+O_{\delta ,K}\left(\sqrt{\frac{\log N}{N}}\right)\right]\ge 1-O_{\delta ,K}(N^{-K}).$$

$$\mathbb{E}{Q}_{11}^2$$

$$\mathbb{E}{Q}_{11}^4$$

$$\mathbb{E}{Q}_{11}^2{Q}_{12}^2$$

$$\mathbb{E}{Q}_{11}^2{Q}_{22}^2$$

$$\mathbb{E}{Q}_{11}{Q}_{12}{Q}_{21}{Q}_{22}$$

$$\bm{Z}\equalscolon\left[\begin{array}[]{ccc}\bm{Z}^{[0,0]}&\bm{Z}^{[0,1]}&\bm{Z}^{[0,2]}\\ \bm{Z}^{[1,0]}&\bm{Z}^{[1,1]}&\bm{Z}^{[1,2]}\\ \bm{Z}^{[2,0]}&\bm{Z}^{[2,1]}&\bm{Z}^{[2,2]}\end{array}\right]=\left[\begin{array}[]{ccc}1&\bm{0}&\mathsf{offdiag}(\bm{M})^{\top}\\ \bm{0}&\bm{M}&\bm{0}\\ \mathsf{offdiag}(\bm{M})&\bm{0}&\bm{Z}^{[2,2]}\end{array}\right].$$

$${\mathsf{pert}}_K(\mathbf{M}):=\left\{\mathbf{A}\in {\mathbb{R}}^d:\mathbf{M}\pm t\mathbf{A}\in K\text{ for all }t>0\text{ sufficiently small }\right\}.$$

Full text

1. 1 Introduction

2. 1.1 Algorithms for the Sherrington-Kirkpatrick Hamiltonian

3. 1.2 Conjectural hardness of certification

4. 1.3 Basic notions of sum-of-squares relaxations

5. 1.4 Prior work on sum-of-squares lower bounds

6. 2 Results

7. 3 Preliminaries

8. 3.1 Glossary: variables, parameters, and constants

9. 3.2 Other notation

10. 3.3 Basic properties of the Montanari-Sen witness

11. 3.4 Moments of $\mathsf{Haar}(\mathcal{O}(N))$

12. 4 Degree 4 pseudomoment construction

13. 4.1 Heuristic 1: evidence from equiangular tight frames

14. 4.2 Heuristic 2: conditional covariance of gaussian matrices

15. 4.3 Precise construction details

16. 5 Conjectural higher-degree extension

17. 6 Proof of positive semidefiniteness: first steps

18. 7 Proof of positive semidefiniteness: main term

19. 7.1 Bounding the cross-term $\bm{T}^{(1)}$

20. 7.2 Bounding the projection term $\bm{T}^{(2)}$: unnormalized case

21. 7.3 Bounding the projection term $\bm{T}^{(2)}$: normalization

22. 7.4 Final main term bound: proof of Lemma 6.1

23. 8 Proof of positive semidefiniteness: correction term

24. Acknowledgements

1 Introduction

1.1 Algorithms for the Sherrington-Kirkpatrick Hamiltonian

This paper concerns convex relaxations of the following optimization problem:

[TABLE]

Since the constraint $x_{i}\in\{\pm 1\}$ may be written $x_{i}^{2}-1=0$, this is a simple instance of quadratically constrained quadratic programming. We are moreover interested in a random setting, where $\bm{W}\in\mathbb{R}^{N\times N}_{\mathsf{sym}}$ is a random matrix drawn from the gaussian orthogonal ensemble (GOE): $W_{ii}\sim\mathcal{N}(0,2/N)$ and $W_{ij}=W_{ji}\sim\mathcal{N}(0,1/N)$, with the $N(N+1)/2$ entries on and above the diagonal distributed independently. We denote this distribution $\bm{W}\sim\mathsf{GOE}(N)$. Under this model, the spectral radius of $\bm{W}$ is of constant order, and the normalization in (1) is such that $\mathbb{E}\mathsf{M}(\bm{W})$ also remains of constant order as $N\to\infty$, as we will describe below.

The problem $\mathsf{M}(\bm{W})$ for general $\bm{W}$ includes the problem of finding maximum cuts in graphs (MaxCut), when $\bm{W}$ is taken to be a graph Laplacian. Karp’s classical result [Kar72] therefore implies that computing $\mathsf{M}(\bm{W})$ is $\mathsf{NP}$-hard in the worst case. The case $\bm{W}\sim\mathsf{GOE}(N)$ is a simple and mathematically elegant example with which we hope to probe the average-case complexity of the same problem, seeking to understand whether the worst-case complexity abates for specific random models of $\bm{W}$.

We are assisted in this task by the rich history of the random optimization problem $\mathsf{M}(\bm{W})$ in statistical physics: up to a change in sign, its value is the ground-state energy of the Sherrington-Kirkpatrick (SK) model, a prominent mean-field model of spin glasses [SK75]. In particular, the asymptotics of its expected value have been well-understood at a non-rigorous level since the seminal work of Parisi [Par79], who developed a system of deep conjectures on the optimization landscape of $\mathsf{M}(\bm{W})$, which, among other results, allowed him to analytically predict the limit

[TABLE]

(Standard results from general gaussian process theory also imply strong concentration around the expectation.) More recently, the computation of this limit has been made mathematically rigorous as well [Pan13a, Pan13b, Tal06].

From the perspective of computer science and optimization, perhaps the more natural random model of $\mathsf{M}(\bm{W})$ is the case where $\bm{W}$ is the adjacency matrix or graph Laplacian of a random graph, which gives randomized instances of MaxCut. A pair of elegant recent works [MS16, DMS*+*17] showed that, in fact, for sparse random graphs this problem is intimately related to the gaussian setting of the SK model: an interpolation argument may be used to control both the true value and the value of a certain simple semidefinite programming relaxation of $\mathsf{M}(\bm{W})$ for sparse random graphs in terms of the SK model.

Thus, whether motivated by the mathematical interest of the GOE and SK model or the application to MaxCut, we are led to ask:

Question 1.1**.**

Under $\bm{W}\sim\mathsf{GOE}(N)$, can $\mathsf{M}(\bm{W})$ be approximated accurately and efficiently?

Of course, knowing the limiting expectation (2) and concentration around this value, it is simple to produce a vacuous algorithm that outputs the value $2\mathsf{P}_{*}$. To capture the difficulty of solving instances of $\mathsf{M}(\bm{W})$ for specific random draws of $\bm{W}$, we must therefore refine our question.

One way to do this is to ask instead:

Question 1.2**.**

Under $\bm{W}\sim\mathsf{GOE}(N)$, can $\bm{x}=\bm{x}(\bm{W})\in\{\pm 1\}^{N}$ be efficiently computed such that $\frac{1}{N}\bm{x}^{\top}\bm{W}\bm{x}\approx 2\mathsf{P}_{*}$?

Recently, assuming a widely-believed conjecture from the spin glass literature, Montanari answered this question in the affirmative in [Mon19]. Montanari’s result followed a similar one on local search in a simpler “random energy model” [ABM18], and used proof techniques related to those proposed by Subag in [Sub18], who addressed the same question in a continuous setting.

Theorem 1.3** (Theorem 2 of [Mon19]).**

Conditional on the conjecture that the Parisi distribution has continuous support at sufficiently low temperature in the SK model,111See Assumption 1 of [Mon19] and the surrounding citations and discussion for further details. for any $\epsilon>0$, there is a polynomial-time algorithm computing $\bm{x}=\bm{x}(\bm{W})\in\{\pm 1\}^{N}$ such that

[TABLE]

Another way to refine our question is to ask rather for certificates of upper bounds on $\mathsf{M}(\bm{W})$:

Question 1.4**.**

Can $c(\bm{W})\in\mathbb{R}$ be efficiently computed with $c(\bm{W})\geq\mathsf{M}(\bm{W})$ and $c(\bm{W})\approx 2\mathsf{P}_{*}$?

(Note that we require $c(\bm{W})\geq\mathsf{M}(\bm{W})$ to hold for every $\bm{W}$; the algorithm is not allowed to “cheat” the random setting by merely outputting a number slightly larger than $2\mathsf{P}_{*}$.) One simple but sub-optimal approach is to form the spectral certificate, which amounts to disregarding the constraint $\bm{x}\in\{\pm 1\}^{N}$ by taking $c(\bm{W})\colonequals\lambda_{\max}(\bm{W})\approx 2$. Recently, Montanari asked222The authors learned of this problem through private communications soon after [MS16] was published. More recently, it was also included in the problem list “AimPL: Phase transitions in randomized computational problems,” available online at http://aimpl.org/phaserandom. whether any certification algorithm could improve on this performance, a problem which, besides modest progress that we will review in the following sections, has since remained open to the best of our knowledge.

Our contribution in this paper is to provide evidence that the spectral certificate is asymptotically optimal by showing that the degree 4 sum-of-squares relaxation, a much more sophisticated convex relaxation, achieves the same performance.

1.2 Conjectural hardness of certification

One step towards making a convincing prediction of whether better-than-spectral certification is possible in the SK model was taken in [BKW20], in which the authors participated. In this work, we first showed that, if efficient certification below 2 were possible for the SK model, then it would be possible to efficiently perform a certain hypothesis testing task in a variant of a spiked matrix model. Then, we provided evidence that this hypothesis testing task should be hard using a method based on the low-degree likelihood ratio. Roughly speaking, this technique takes low-degree polynomials as a proxy for all polynomial-time testing statistics and measures their performance in a convenient smoothed sense, which allows the optimal low-degree polynomial statistic to be identified and analyzed using an orthogonal polynomial decomposition.

This suggests the following conjecture, which would hold conditional on another, quite broad conjecture of [HS17, Hop18] that the low-degree likelihood ratio analysis is correct for a large class of hypothesis testing problems.

Conjecture 1.5**.**

For any $\epsilon>0$, there does not exist a polynomial-time certification algorithm for $\mathsf{M}(\bm{W})$ such that $c(\bm{W})\leq 2-\epsilon$ with high probability.

Unfortunately, though the low-degree likelihood ratio method predicts many known computational thresholds in random problems correctly, at the moment it is only known to imply rather weak lower bounds against specific algorithms—either only lower bounds in expectation or a smoothed $L^{2}$ sense, or high-probability lower bounds under quite restrictive assumptions (see, e.g., the recent survey [KWB19] by the authors). In search of further evidence of hardness of certification, we therefore consider concrete algorithms and analyze their performance directly.

1.3 Basic notions of sum-of-squares relaxations

The main algorithmic approach for certifying bounds on a problem like $\mathsf{M}(\bm{W})$ is to form convex relaxations that may be solved efficiently by standard convex optimization techniques. First, note that, defining the cut polytope

[TABLE]

we may rewrite $\mathsf{M}(\bm{W})$ as a linear optimization problem over this set of matrices,

[TABLE]

Though $\mathscr{C}^{N}$ is a convex set, it is complex to describe [DL09], and in particular does not admit a polynomial-time separation oracle unless $\mathsf{P}=\mathsf{NP}$ (by the same result of [Kar72] mentioned before). We thus pursue the idea of expanding $\mathscr{C}^{N}$ to a larger convex set that may be described more simply, and over which convex optimization is tractable.

Specifically, we will study the performance of semidefinite programming (SDP) relaxations of $\mathsf{M}(\bm{W})$. Perhaps the simplest of these is based on the inclusion of sets

[TABLE]

Replacing $\mathscr{C}^{N}$ with $\mathscr{E}^{N}_{\mathsf{spec}}$ in the definition of $\mathsf{M}(\bm{W})$ just computes $\lambda_{\max}(\bm{W})$, expressing the spectral certificate as an SDP relaxation. Thus one consequence of Conjecture 1.5 is that this naive relaxation is optimal among the wide variety of SDP relaxations that may be applied to $\mathsf{M}(\bm{W})$.

One broad and successful framework for SDP relaxation of optimization problems through which one might hope to find an improvement is the sum-of-squares (SOS) hierarchy of relaxations [Las01, Lau09, BPT12, BS14]. This generates a sequence of convex sets we will denote by $\mathscr{E}_{d}^{N}$, indexed by a parameter $d$, an even natural number called the degree, which satisfy [Lau03, FSP16] the strict inclusions

[TABLE]

Moreover, $\mathscr{E}_{d}^{N}$ is a projection of an affine slice of the positive semidefinite cone of $N^{d/2}\times N^{d/2}$ real symmetric matrices, whereby optimization over $\mathscr{E}_{d}^{N}$ may be written as an SDP.

Unfortunately, though in this SDP both the dimension of the decision variable and the number of constraints scale polynomially with $N$, and even if we assume (as will be the case in our application) that the SDP coefficients may be well-approximated with an encoding in polynomially many bits (i.e., are of size bounded by $\exp(\mathsf{poly}(N))$), it still need not be the case that the SDP can be solved to small additive error in polynomial time, as O’Donnell has pointed out [O’D17]. The key point is that one must further ensure that there exists an optimizer of the SDP whose entries are also of size bounded by $\exp(\mathsf{poly}(N))$. Fortunately, the work [RW17] studied this issue for sum-of-squares relaxations of many discrete problems, including our setting of unconstrained optimization over Boolean variables, and showed that this condition is in fact satisfied (see their Corollary 9). Thus optimization over $\mathscr{E}_{d}^{N}$ may indeed be performed in time $N^{O(d)}$ with, for instance, the ellipsoid algorithm.

To give a concrete formulation of this SDP, we now describe $\mathscr{E}_{d}^{N}$ in terms of the pseudomoment interpretation of SOS optimization (as derived from the general formulation of SOS in, e.g., [Lau03]). Below we adopt the useful notations of $\binom{[N]}{k}$ and $\binom{[N]}{\leq k}$ for the sets of subsets of $[N]$ having size $k$ and size at most $k$ (including the empty set), respectively. We also use the standard notation $S\triangle T\colonequals(S\setminus T)\cup(T\setminus S)$ for the symmetric difference of the sets $S$ and $T$.

Definition 1.6**.**

$\mathscr{E}_{d}^{N}\subset\mathbb{R}^{N\times N}_{\mathsf{sym}}$* is the set of matrices $\bm{M}$ such that there exists $\bm{Z}\in\mathbb{R}^{\binom{[N]}{\leq d/2}\times\binom{[N]}{\leq d/2}}$ having*

[TABLE]

and satisfying the following properties:

$\bm{Z}\succeq\bm{0}$. 2. 2.

$Z_{ST}$* only depends on $S\triangle T$.* 3. 3.

$Z_{ST}=1$* whenever $S\triangle T=\varnothing$.*

In this case, we say $\bm{Z}$ is a degree $d$ pseudomoment matrix for the constraint polynomials $\{x_{i}^{2}-1:i\in[N]\}\subset\mathbb{R}[x_{1},\dots,x_{N}]$, which extends $\bm{M}$.333Usually, a degree $d$ pseudomoment matrix must be indexed by all monomials in $x_{1},\dots,x_{N}$ of degree at most $d/2$; however, in our case, the constraint $x_{i}^{2}=1$ ensures that the pseudomoments of multilinear monomials fully determine the pseudomoment matrix.

For the sake of brevity, we will simply refer to such $\bm{Z}$ as a degree $d$ pseudomoment matrix, since we only study optimization over $\{\pm 1\}^{N}$.

Definition 1.7**.**

The degree $d$ sum-of-squares relaxation of $\mathsf{M}(\bm{W})$ is

[TABLE]

In addition to Montanari’s general question on certifying bounds on $\mathsf{M}(\bm{W})$ mentioned before, Jain, Risteski, and Koehler have independently posed the more specific question of determining the asymptotic value of $\mathsf{SOS}_{d}(\bm{W})$ when $\bm{W}\sim\mathsf{GOE}(N)$ in [JKR19].

One further simplification of the above setup will also be useful to take into account.

Definition 1.8**.**

We call $\bm{Z}\in\mathbb{R}^{\binom{[N]}{\leq d/2}\times\binom{[N]}{\leq d/2}}$ a reduced degree $d$ pseudomoment matrix if it satisfies all of the conditions of Definition 1.6, as well as the following additional condition:

$Z_{ST}=0$* whenever $|S\triangle T|$ is odd.*

As we show below, because of the invariance of both the constraints $x_{i}^{2}-1=0$ and the objective function $\bm{x}^{\top}\bm{W}\bm{x}$ under the map $\bm{x}\mapsto-\bm{x}$, $\mathscr{E}_{d}^{N}$ may be equivalently defined in terms of reduced pseudomoment matrices.

Proposition 1.9**.**

If there exists a degree $d$ pseudomoment matrix extending $\bm{M}$, then there exists a reduced degree $d$ pseudomoment matrix extending $\bm{M}$.

Proof.

It suffices to show that if $\bm{Z}^{(0)}$ is a degree $d$ pseudomoment matrix, then the matrix $\bm{Z}^{(1)}$ of the same shape with entries $Z^{(1)}_{ST}\colonequals(-1)^{|S\triangle T|}Z^{(0)}_{ST}$ is also a degree $d$ pseudomoment matrix, since then $\frac{1}{2}\bm{Z}^{(0)}+\frac{1}{2}\bm{Z}^{(1)}$ is a reduced degree $d$ pseudomoment matrix whose minor indexed by $\binom{[N]}{1}$ equals that of $\bm{Z}^{(0)}$.

Clearly all of the linear constraints on $\bm{Z}^{(1)}$ are satisfied, so it suffices to check positive semidefiniteness. For this, if $\bm{x}^{(1)}\in\mathbb{R}^{\binom{[N]}{\leq d/2}}$, let us define $\bm{x}^{(0)}\in\mathbb{R}^{\binom{[N]}{\leq d/2}}$ by $x^{(0)}_{S}=(-1)^{|S|}x^{(0)}_{S}$. Then, since $(-1)^{|S|+|T|}=(-1)^{|S\triangle T|}$, we have $\bm{x}^{(1)^{\top}}\bm{Z}^{(1)}\bm{x}^{(1)}=\bm{x}^{(0)^{\top}}\bm{Z}^{(0)}\bm{x}^{(0)}\geq 0$, completing the proof. ∎

We will only study degree 2 and degree 4 pseudomoment matrices in detail, so we give more concrete versions of the above conditions for those cases.

Proposition 1.10**.**

$\mathscr{E}_{2}^{N}=\{\bm{M}\in\mathbb{R}^{N\times N}_{\mathsf{sym}}:\bm{M}\succeq\bm{0},M_{ii}=1\text{ for all }i\in[N]\}.$**

Proposition 1.11**.**

Let $\bm{Z}\in\mathbb{R}^{(N(N+1)/2+1)\times(N(N+1)/2+1)}_{\mathsf{sym}}$, with the row and column indices of $\bm{Z}$ identified with $\binom{[N]}{\leq 2}$, ordered first by size and then in lexicographical order.444For instance, $\binom{[3]}{\leq 2}$ is ordered as $\varnothing,\{1\},\{2\},\{3\},\{1,2\},\{1,3\},\{2,3\}$. Then, $\bm{Z}$ is a reduced degree 4 pseudomoment matrix if and only if the following conditions hold:

$\bm{Z}\succeq\bm{0}$. 2. 2.

$Z_{\varnothing\varnothing}=Z_{\{i\}\{i\}}=Z_{\{i,j\}\{i,j\}}=1$* for all distinct $i,j\in[N]$.* 3. 3.

$Z_{\{i\}\varnothing}=Z_{\{i,j\}\{j\}}=0$* for all distinct $i,j\in[N]$.* 4. 4.

$Z_{\{i,j\}\{i,k\}}=Z_{\{j,k\}\varnothing}=Z_{\{j\}\{k\}}$* for all distinct $i,j,k\in[N]$.* 5. 5.

$Z_{\{i,j\}\{k,\ell\}}$* is invariant under permutations of the indices $i,j,k,\ell$.*

1.4 Prior work on sum-of-squares lower bounds

The Montanari-Sen degree 2 lower bound

The only previous result on SOS relaxations of $\mathsf{M}(\bm{W})$ under $\bm{W}\sim\mathsf{GOE}(N)$ that we are aware of is the following result of Montanari and Sen [MS16], which establishes hardness of certification for degree 2 SOS.

Theorem 1.12** (Theorem 5(a) of [MS16]).**

Let $\epsilon>0$. Then,

[TABLE]

The mechanics of this result will be crucial for proving ours, so let us also review how to construct feasible points achieving this bound. First, fix a parameter $\delta\in(0,1)$, and set $r=r(N)=\delta N$.555Following [MS16], we assume for the sake of simplicity that $\delta N$ is an integer; recovering the same results for $r=\lfloor\delta N\rfloor$ is tedious but straightforward. Then, given $\bm{W}\sim\mathsf{GOE}(N)$, let $V\subset\mathbb{R}^{N}$ be the subspace spanned by the $r(N)$ eigenvectors of $\bm{W}$ having the largest eigenvalues, and let $\bm{P}$ be the orthogonal projector to $V$. Let $\bm{D}\in\mathbb{R}^{N\times N}$ be the diagonal matrix with entries $D_{ii}=P_{ii}$, and define

[TABLE]

(Note that $D_{ii}=0$ if and only if $\bm{e}_{i}\in V^{\perp}$, which almost surely does not occur.) Then, $\bm{M}^{(\delta)}(\bm{W})\in\mathscr{E}_{2}^{N}$ for any $\delta\in(0,1)$ almost surely, and for each $\epsilon>0$ there exists $\delta\in(0,1)$ such that

[TABLE]

We call $\bm{M}^{(\delta)}$ the Montanari-Sen witness. This construction will be the basis of ours; indeed, we will show that only a small correction is necessary to make this witness feasible for the degree 4 SOS relaxation as well.

Simultaneous work on degree 4 lower bounds

After preparing an earlier version of this manuscript, we learned of the concurrent work [MRX19], the fruit of a parallel research effort in which the same result (Corollary 2.2, stated in the following section) as ours is proved. Their construction uses the pseudocalibration sum-of-squares heuristic introduced in [BHK*+*19], while our construction is based on more geometric and problem-specific considerations of constraints on the pseudomoment extension matrices described above. We remark that one important advantage of their work is that its analysis is general enough to apply immediately to problems besides the SK model, such as MaxCut on sparse random graphs.

On the other hand, the output of pseudocalibration in this situation appears to be quite complicated: while the degree 4 pseudomoments constructed in [MRX19], like ours, are low-degree polynomials in the entries of $\bm{M}$, they are not stated explicitly, and are sums of many more “graphical” terms, simple polynomials in the entries of $\bm{M}$ described by graphs labelled by matrix indices. For instance, while the formula we will derive for our degree 4 pseudomoments includes just two types of such polynomials (see our Section 4), the analysis of the pseudomoments of [MRX19] involves at least 20 types (see their Section 3). Thus, though it is possible that the degree 4 pseudomoment matrices constructed in our work are close to those constructed in [MRX19] (in spectral norm, for example), our approach gives a more explicit and condensed description of a suitable pseudomoment construction.

Furthermore, while both this paper and [MRX19] use the idea of an “approximate Cholesky decomposition” of the pseudomoment matrix (also present in earlier works such as [BHK*+*19]) for the proofs of positive semidefiniteness, we give in Sections 4.2 and 5 an intuitive probabilistic meaning to the Cholesky factors arising in this argument. To the best of our knowledge, this is a novel interpretation, which we believe may shed new light on the structure of pseudomoment matrices and simplify some of the ad hoc technicalities arising in the proofs of their positive semidefiniteness.

2 Results

Our main result establishes hardness of certification for degree 4 SOS, by showing that a minor variant of the Montanari-Sen witness admits a degree 4 extension.

Theorem 2.1**.**

Let $\alpha,\delta\in(0,1)$, and let $\bm{M}^{(\delta)}$ be as in Theorem 1.12. Define

[TABLE]

Then,

[TABLE]

The theorem says that an arbitrarily small adjustment in the direction the identity matrix (the barycenter of the vertices of the cut polytope) suffices to make the degree 2 Montanari-Sen primal witness admit a degree 4 extension with high probability.666The specific choice of the identity matrix is probably not essential, but is convenient because, as we will see, the degree 4 extension of the identity matrix has an especially simple spectral structure. In Conjecture 5.2, we will also propose that the same membership holds with high probability for any SOS relaxation $\mathscr{E}_{d}^{N}$ with constant degree $d$ as $N\to\infty$.

Since the degree 2 part of our construction is a simple modification of the Montanari-Sen witness, it is straightforward to apply Theorem 1.12 and obtain a lower bound for the degree 4 SOS objective.

Corollary 2.2**.**

Let $\epsilon>0$. Then,

[TABLE]

Proof.

By Theorem 1.12, take $\delta\in(0,1)$ such that $\frac{1}{N}\langle\bm{M}^{(\delta)}(\bm{W}),\bm{W}\rangle\geq 2-\epsilon/3$ with high probability. Let $\alpha\in(0,1)$ be small that $(1-\alpha)(2-\epsilon/3)\geq 2-2\epsilon/3$. By Theorem 2.1, with high probability,

[TABLE]

The random variable $\mathrm{Tr}(\bm{W})$ has law $\mathcal{N}(0,2)$, so with high probability the last term is smaller than $\epsilon/3$, and the result follows. ∎

Organization

The remainder of the paper gives the proof of Theorem 2.1, and conjectures an extension of the same proof technique to higher degree SOS relaxations. In Section 3 we review some preliminary facts and notations. In Section 4 we give the motivation and precise statement of our construction of a matrix that is with high probability a degree 4 pseudomoment matrix extending $\bm{M}^{(\alpha,\delta)}(\bm{W})$. In Section 5, we present a natural conjecture for how this construction can be extended to higher degrees. Finally, in Sections 6, 7, and 8, we prove that our construction from Section 4 indeed furnishes a valid degree 4 pseudomoment matrix with high probability.

3 Preliminaries

3.1 Glossary: variables, parameters, and constants

For reference, we summarize various symbols that will appear throughout. The following are the general scalar parameters involved.

• •

$N$ is a parameter indicating the size of $\bm{W}\sim\mathsf{GOE}(N)$ in the problem statement.

• •

$\delta\in(0,1)$ is a fixed parameter not depending on $N$.

• •

$r=r(N)\colonequals\delta N$, which we assume for the sake of simplicity is an integer. This gives the rank of the Montanari-Sen witness that we extend (before modifications that make it have full rank)

• •

$\alpha\in(0,1)$ is another fixed parameter not depending on $N$ or $\delta$.

• •

$K$ is a constant appearing in concentration inequalities, giving polynomial rates of decay of probabilities of the form $N^{-K}$. All of the concentration inequalities where $K$ appears hold with any choice of $K>0$, but the other constants appearing in those results depend on both $K$ and $\delta$. Thus a typical inequality will take the form

[TABLE]

One may think of any concrete choice, e.g. $K=100$, throughout.

The following are vectors and matrices associated to the Montanari-Sen construction applied to a specific instance $\bm{W}$.

• •

$\bm{V}\in\mathbb{R}^{r\times N}$ is the matrix having the top $r$ (unit norm) eigenvectors of $\bm{W}$ as its rows.

• •

$\bm{v}_{1},\dots,\bm{v}_{N}\in\mathbb{R}^{r}$ are the columns of $\bm{V}$ (not eigenvectors of $\bm{W}$).

• •

$\bm{P}=\bm{V}^{\top}\bm{V}$ is the orthogonal projector to the top $r$-dimensional eigenspace of $\bm{W}$, and also the Gram matrix of the $\bm{v}_{i}$.

• •

$\widehat{\bm{v}}_{i}=\bm{v}_{i}/\|\bm{v}_{i}\|_{2}$.

• •

$\bm{D}\in\mathbb{R}^{N\times N}$ is a diagonal matrix with $D_{ii}=\|\bm{v}_{i}\|_{2}^{2}=P_{ii}$.

• •

$\widehat{\bm{V}}=\bm{V}\bm{D}^{-1/2}\in\mathbb{R}^{r\times N}$ is the matrix having the $\widehat{\bm{v}}_{i}$ as its columns.

• •

$\bm{M}=\bm{M}^{(\delta)}=\widehat{\bm{V}}^{\top}\widehat{\bm{V}}=\bm{D}^{-1/2}\bm{P}\bm{D}^{-1/2}$ is the Montanari-Sen witness, and also the Gram matrix of the $\widehat{\bm{v}}_{i}$. For the sake of brevity, we will often drop the $(\delta)$ superscript, as $\delta$ will be a constant carried throughout.

3.2 Other notation

Linear algebra

The identity matrix in dimension $n$ is denoted $\bm{I}_{n}$, and the all-ones vector is denoted $\bm{1}_{n}$. The Frobenius or entrywise inner product of matrices having the same shape is denoted $\langle\bm{A},\bm{B}\rangle=\mathrm{Tr}(\bm{A}^{\top}\bm{B})=\sum_{i,j}A_{ij}B_{ij}$. The Hadamard or entrywise product of matrices having the same shape is denoted $\bm{A}\circ\bm{B}$ and has entries $(\bm{A}\circ\bm{B})_{ij}=A_{ij}B_{ij}$. Hadamard powers of a matrix are denoted $\bm{A}^{\circ k}$. The matrix operator norm is denoted $\|\bm{A}\|_{\mathsf{op}}=\max_{\|\bm{x}\|_{2}=\|\bm{y}\|_{2}=1}\bm{x}^{\top}\bm{A}\bm{y}$. The vectorized supremum norm is denoted $\|\bm{A}\|_{\ell^{\infty}}=\max_{i,j}|A_{ij}|$. The matrix Frobenius norm is denoted $\|\bm{A}\|_{F}=\langle\bm{A},\bm{A}\rangle^{1/2}$. The group of $N\times N$ orthogonal matrices is denoted $\mathcal{O}(N)$. The Stiefel manifold of $r\times N$ matrices with orthonormal rows is denoted $\mathsf{Stief}(N,r)$.

Vectorization

We describe several ways of vectorizing symmetric matrices. For $\bm{A}\in\mathbb{R}^{n\times n}_{\mathsf{sym}}$, we let $\mathsf{diag}(\bm{A})\in\mathbb{R}^{n}$ and $\mathsf{offdiag}(\bm{A})\in\mathbb{R}^{n(n-1)/2}$ be the vectorized diagonal and strict upper triangle of $\bm{A}$, respectively, with index sets ordered lexicographically. These two vectors determine a symmetric matrix completely. It will also be useful to define an isometry between $\mathbb{R}^{n\times n}_{\mathsf{sym}}$ endowed with the Frobenius inner product and $\mathbb{R}^{n(n+1)/2}$ endowed with the ordinary Euclidean inner product. This is given by

[TABLE]

which indeed satisfies $\langle\mathsf{isovec}(\bm{A}),\mathsf{isovec}(\bm{B})\rangle=\langle\bm{A},\bm{B}\rangle$. We denote $\bm{1}_{\mathsf{diag}}\colonequals\mathsf{isovec}(\bm{I}_{n})$ when the dimension $n$ is clear from context, since this is the indicator vector of the diagonal matrix indices.

Probability

We denote the relation of two random variables $X,Y$ having the same law by $X\stackrel{{\scriptstyle\mathrm{(d)}}}{{=}}Y$. The Haar measure on the orthogonal group or Stiefel manifold is denoted $\mathsf{Haar}(\mathcal{O}(N))$ or $\mathsf{Haar}(\mathsf{Stief}(N,r))$, respectively, and refers to Haar measure with respect to the action of $\mathcal{O}(N)$ by right multiplication in the latter case. We write $\mathsf{Unif}(\SS^{N-1})$ for the Haar measure on vectors of the unit sphere under the action of $\mathcal{O}(N)$ by left multiplication, equivalent to $\mathsf{Haar}(\mathsf{Stief}(N,1))$ (up to transposition).

3.3 Basic properties of the Montanari-Sen witness

It will be useful to establish some preliminary bounds on and distributional properties of the Montanari-Sen construction described in Theorem 1.12.

First, we use an elegant geometric argument mentioned in [MS16] (but which seems to be well-known folklore) to obtain bounds on the entries of $\bm{P}$.

Proposition 3.1**.**

For all $K>0$,

[TABLE]

Proof.

For the diagonal entries, $P_{ii}=\|\bm{P}\bm{e}_{i}\|_{2}^{2}\stackrel{{\scriptstyle\mathrm{(d)}}}{{=}}\|\bm{P}_{0}\bm{v}\|_{2}^{2}$ for $\bm{v}\sim\mathsf{Unif}(\SS^{N-1})$ and $\bm{P}_{0}$ the orthogonal projector onto $\mathsf{span}(\bm{e}_{1},\dots,\bm{e}_{r})$. Letting $\bm{g}\sim\mathcal{N}(\bm{0},\bm{I}_{N})$, we have $\bm{v}\stackrel{{\scriptstyle\mathrm{(d)}}}{{=}}\bm{g}/\|\bm{g}\|_{2}$. Writing $\bm{g}^{(r)}$ and $\bm{g}^{(N-r)}$ for the first $r$ and last $N-r$ coordinates of $\bm{g}$ respectively, we then have

[TABLE]

where $\|\bm{g}^{(r)}\|_{2}^{2}$ and $\|\bm{g}^{(N-r)}\|_{2}^{2}$ are distributed as independent $\chi^{2}$ random variables with $r$ and $N-r$ degrees of freedom respectively. The result then follows from the concentration inequalities of [LM00] for $\chi^{2}$ random variables.

For the off-diagonal entries, likewise $P_{ij}=\langle\bm{P}\bm{e}_{i},\bm{P}\bm{e}_{j}\rangle\stackrel{{\scriptstyle\mathrm{(d)}}}{{=}}\langle\bm{P}_{0}\bm{v},\bm{P}_{0}\bm{w}\rangle$, where now $(\bm{v},\bm{w})$ are a Haar-distributed two-dimensional orthonormal frame. If we draw $\bm{g},\bm{h}\sim\mathcal{N}(\bm{0},\bm{I}_{N})$ independently, then by performing two steps of Gram-Schmidt orthonormalization,

[TABLE]

Thus computing as before we find

[TABLE]

To control these quantities, we first bound

[TABLE]

In this expression, every $\ell^{2}$ norm may be controlled by the concentration inequalities of [LM00] as before, and every inner product may be controlled by observing that, when $\bm{g}$ and $\bm{h}$ are independent standard gaussian vectors, then $\langle\bm{g},\bm{h}\rangle\stackrel{{\scriptstyle\mathrm{(d)}}}{{=}}\|\bm{g}\|_{2}h_{1}$, thus reducing the task of controlling the inner product to controlling an $\ell^{2}$ norm. Applying these bounds then gives the result. ∎

The following related results for $\bm{M}$ follow directly.

Corollary 3.2**.**

For all $K>0$,

[TABLE]

Proof.

We have $M_{ij}=P_{ij}/\sqrt{P_{ii}P_{jj}}$, so the result follows from combining Proposition 3.1 applied to several entries. ∎

Corollary 3.3**.**

For all $K>0$,

[TABLE]

Proof.

$\bm{M}=\bm{D}^{-1/2}\bm{P}\bm{D}^{-1/2}$, so $\|\bm{M}\|_{\mathsf{op}}\leq\|\bm{D}^{-1/2}\|_{\mathsf{op}}^{2}=(\min_{i\in[N]}P_{ii})^{-1}$, and the result then follows by Proposition 3.1. ∎

3.4 Moments of $\mathsf{Haar}(\mathcal{O}(N))$

The following gives the low-degree moments of Haar-distributed orthogonal matrices.

Proposition 3.4** (Lemma 9 of [CM08]).**

Let $\bm{Q}\sim\mathsf{Haar}(\mathcal{O}(N))$. The moment $\mathbb{E}\prod_{k=1}^{d}Q_{i_{k}j_{k}}$ is zero if any index occurs an odd number of times among either the $i_{k}$ or $j_{k}$. The non-zero degree 2 and 4 moments are given by

[TABLE]

4 Degree 4 pseudomoment construction

In this section we outline the main idea in the proof of Theorem 2.1, detailing the construction of a suitable degree 4 pseudomoment matrix and reducing Theorem 2.1 to verifying the positive semidefiniteness of this matrix.

First, we give two heuristic lines of reasoning supporting the construction we propose. The first derives a formula for the degree 4 pseudomoment extension of a highly structured collection of degree 2 pseudomoments and assumes that this formula may be transferred verbatim to the random case. The second recovers the same formula after applying some simplifying heuristics to a probabilistic argument.

As a preliminary, let us remark that, per Proposition 1.9, we may restrict our attention to reduced degree 4 pseudomoment matrices. If $\bm{Z}$ is a reduced degree 4 pseudomoment matrix extending $\bm{M}$, and we divide $\bm{Z}$ into blocks indexed by $\binom{[N]}{0}$, $\binom{[N]}{1}$, and $\binom{[N]}{2}$, where $\bm{Z}^{[i,j]}$ is the block with rows indexed by $\binom{[N]}{i}$ and columns indexed by $\binom{[N]}{j}$, then only the block $\bm{Z}^{[2,2]}$ is not determined by the properties of being reduced and extending $\bm{M}$:

[TABLE]

Thus in the sequel we will be justified in saying that an extension $\bm{Z}$ of $\bm{M}$ is specified just by a particular choice of $\bm{Z}^{[2,2]}$.

4.1 Heuristic 1: evidence from equiangular tight frames

In this section, we review a result from the works [BK18, BK19] of the authors’, which derived an explicit description of degree 4 extensions of degree 2 pseudomoment matrices for some very structured special cases, that resemble the Montanari-Sen witness in that their degree 2 pseudomoment matrices are constant multiples of orthogonal projectors.

The extra structure that allows this description of the degree 4 pseudomoments to be derived in closed form is described by the following notions from finite frame theory.

Definition 4.1**.**

A collection of vectors $\widehat{\bm{v}}_{1},\dots,\widehat{\bm{v}}_{N}\in\mathbb{R}^{r}$ forms a unit norm tight frame (UNTF) if the following conditions hold.

(Unit Norm) $\|\widehat{\bm{v}}_{i}\|_{2}=1$ for all $i\in[N]$. 2. 2.

(Tight Frame) $\sum_{i=1}^{N}\widehat{\bm{v}}_{i}\widehat{\bm{v}}_{i}^{\top}=\frac{N}{r}\bm{I}_{r}$.

They moreover form an equiangular tight frame (ETF) if the following additional condition holds.

(Equiangular) There is $\mu\in[0,1]$ such that $|\langle\widehat{\bm{v}}_{i},\widehat{\bm{v}}_{j}\rangle|=\mu$ whenever $i\neq j$.

ETFs are rare and combinatorially structured objects [STDHJ07, CRT08, FM15], which do not seem a priori related to the SK problem and the Montanari-Sen witness. However, it turns out that studying the degree 4 extensions of ETFs gives useful insight into the correct construction of pseudomoments even in the random case.

In general, we showed in [BK18] that degree 4 extensions are related to the following notion from convex geometry.

Definition 4.2**.**

Let $K\subseteq\mathbb{R}^{d}$ be a closed convex set. For $\bm{M}\in K$, the perturbation of $\bm{M}$ in $K$ is the linear subspace

[TABLE]

Equivalently, if $H$ is the affine hull of the minimal face of $K$ that $\bm{M}$ belongs to, then $\mathsf{pert}_{K}(\bm{M})=H-\bm{M}$.

The relevant case for degree 4 extensions is $K=\mathscr{E}_{2}^{N}$. In this case, the following theorem characterizes the perturbation subspace.

Proposition 4.3** (Theorem 1(a) of [LT94]).**

Let $\bm{M}\in\mathscr{E}_{2}^{N}$ have $\mathsf{rank}(\bm{M})=r$ and $\bm{M}=\widehat{\bm{V}}^{\top}\widehat{\bm{V}}$ for $\widehat{\bm{V}}\in\mathbb{R}^{r\times N}$ with unit vector columns $\widehat{\bm{v}}_{1},\dots,\widehat{\bm{v}}_{N}$. Then,

[TABLE]

For ETFs, a degree 4 extension, when any exists, is given explicitly in both spectral and entrywise terms as follows.

Theorem 4.4** (Theorem 2.19 of [BK18]).**

Let $\bm{M}\in\mathscr{E}_{2}^{N}$ be the Gram matrix of an ETF of $N$ vectors in $\mathbb{R}^{r}$. Then, $\bm{M}\in\mathscr{E}_{4}^{N}$ if and only if $N<\frac{r(r+1)}{2}$. In this case, a degree 4 extension $\bm{Z}\in\mathbb{R}^{(N(N+1)/2+1)\times(N(N+1)/2+1)}_{\mathsf{sym}}$ is given by

[TABLE]

and the block structure given in (31). The entries of this matrix are given by

[TABLE]

(The spectral description will be of interest later, to draw a connection to the second heuristic presented in the following section.)

The Montanari-Sen witness $\bm{M}$ is close to a constant multiple of a projection matrix, since by Proposition 3.1 all entries of the normalizing diagonal matrix $\bm{D}$ are close to $\delta$, so $\bm{M}$ is a “near-UNTF Gram matrix.” Also, the off-diagonal entries of $\bm{M}$ are inner products of random unit vectors $\widehat{\bm{v}}_{i}$, which it is reasonable to think are weakly dependent, whereby $\bm{M}$ should moreover behave like “an ETF in expectation.”

Thus to guess a degree 4 extension for the Montanari-Sen witness $\bm{M}$, we may be justified in simply trying to apply the combinatorial ETF construction directly. Since we take $r=\delta N$ with $r,N\to\infty$, we may also simplify the leading coefficients to their asymptotic values, which gives the prediction

[TABLE]

4.2 Heuristic 2: conditional covariance of gaussian matrices

We next show another, perhaps more principled argument through which we arrive at the same prediction of degree 4 pseudomoments. Let us suppose that $\bm{M}$ is exactly the Gram matrix of a UNTF, i.e. $\bm{M}=\delta^{-1}\bm{P}=\delta^{-1}\bm{V}^{\top}\bm{V}$ and $\mathsf{diag}(\bm{P})=\delta\bm{1}_{N}$. As mentioned above, this is approximately the case for the Montanari-Sen witness, but our heuristic derivation is much simplified if the correction by the diagonal matrix $\bm{D}$ to form the actual Montanari-Sen witness may be reduced to a constant scaling.

For the computations to come it is more convenient to view $\bm{Z}^{[2,2]}$ as being embedded in a larger pseudomoment matrix, indexed by pairs $(i,j)\in[N]^{2}$. We denote this matrix by $\bm{Z}^{\mathsf{mult}[2,2]}$, and, following the pseudomoment framework discussed before, set $Z^{\mathsf{mult}[2,2]}_{(i,j)(k,\ell)}$ to be the “pseudoexpectation” of the monomial $x_{i}x_{j}x_{k}x_{\ell}$. In terms of $\bm{Z}^{[2,2]}$ and $\bm{M}$, these entries are

[TABLE]

We note that if $\bm{Z}$ is positive semidefinite then so is $\bm{Z}^{\mathsf{mult}[2,2]}$, since, up to removing repeated rows and columns, the latter is equal to the principal minor of the former indexed by $\{\varnothing\}\cup\binom{[N]}{2}$.

Now, we view $\bm{Z}^{\mathsf{mult}[2,2]}$, a positive semidefinite $N^{2}\times N^{2}$ matrix, as the degree 2 moment matrix of the entries of a gaussian random matrix: we suppose there exists some $\bm{A}\in\mathbb{R}^{N\times N}$ with random jointly gaussian entries such that

[TABLE]

We then design $\bm{A}$ so that $\bm{Z}^{\mathsf{mult}[2,2]}$ automatically satisfies some of the necessary constraints, and hope that the remaining constraints will be approximately satisfied as well.

We begin with a matrix $\bm{A}^{(0)}$ having a canonical gaussian distribution for symmetric matrices, the GOE, suitably rescaled to allow us a normalizing degree of freedom later: $A_{ii}^{(0)}\sim\mathcal{N}(0,2\sigma^{2})$ and $A_{ij}^{(0)}=A_{ji}^{(0)}\sim\mathcal{N}(0,\sigma^{2})$. Taking $\bm{A}$ to be symmetric already ensures some of the symmetry conditions that $\bm{Z}^{\mathsf{mult}[2,2]}$ must satisfy. Next, we take $\bm{A}$ to have the distribution of $\bm{A}^{(0)}$, conditional on the following two properties:

$(\bm{I}_{N}-\bm{P})\bm{A}=0$. 2. 2.

$A_{ii}=1$ for all $i\in[N]$.

Property 1 ensures that any symmetric matrix formed from $\bm{Z}^{\mathsf{mult}[2,2]}$ by “freezing” one index pair $(i,j)$ and letting the other index pair vary, $\bm{F}^{(i,j)}\colonequals(Z_{(i,j)(k,\ell)}^{\mathsf{mult}[2,2]})_{k,\ell=1}^{N}$, has row (or column) space contained in that of $\bm{M}$. Every $\bm{Z}^{\mathsf{mult}[2,2]}$ constructed as above must have this property, because it has a principal minor of the form

[TABLE]

whose positive semidefiniteness gives this condition on $\bm{F}^{(i,j)}$. Property 2 ensures that $Z_{(i,i)(k,\ell)}^{\mathsf{mult}[2,2]}$ does not depend on the index $i$, which is required by our definition of $\bm{Z}^{\mathsf{mult}[2,2]}$ above, and reflects the application of the polynomial constraint $x_{i}^{2}=1$ to the monomial $x_{i}^{2}x_{k}x_{\ell}$.

What is the law of the resulting gaussian matrix $\bm{A}$? Conditioning on Property 1 yields the law of $\bm{P}\bm{A}^{(0)}\bm{P}=\bm{V}(\bm{V}^{\top}\bm{A}^{(0)}\bm{V})\bm{V}^{\top}$. By rotational invariance of the GOE, the inner matrix $\bm{V}^{\top}\bm{A}^{(0)}\bm{V}\equalscolon\bm{A}^{(1)}\in\mathbb{R}^{r\times r}_{\mathsf{sym}}$ has the same law as the upper left $r\times r$ block of $\bm{A}^{(0)}$, i.e., a smaller GOE matrix with the same variance scaling of $\sigma^{2}$.

Next, we condition on Property 2, or equivalently condition $\bm{A}^{(1)}$ on having $\bm{v}_{i}^{\top}\bm{A}^{(1)}\bm{v}_{i}=\langle\bm{v}_{i}\bm{v}_{i}^{\top},\bm{A}^{(1)}\rangle=1$. $\bm{A}^{(1)}$ has the law of $\mathsf{isovec}^{-1}(\bm{a})$ for a gaussian vector $\bm{a}\sim\mathcal{N}(\bm{0},2\sigma^{2}\bm{I}_{r(r+1)/2})$. Since $\mathsf{isovec}$ is an isometry, we may equivalently condition $\bm{a}$ on $\langle\bm{a},\mathsf{isovec}(\bm{v}_{i}\bm{v}_{i}^{\top})\rangle=1$ for each $i\in[N]$. By basic properties of gaussian conditioning, the resulting law is

[TABLE]

where $\bm{P}^{\circ 2}$ is the Gram matrix of the $\mathsf{isovec}(\bm{v}_{i}\bm{v}_{i}^{\top})$ or equivalently the entrywise square of $\bm{P}$, and $\widetilde{\bm{P}}$ is the orthogonal projector to the span of the $\mathsf{isovec}(\bm{v}_{i}\bm{v}_{i}^{\top})$. Let $\bm{A}^{(2)}$ be a matrix with the law of $\mathsf{isovec}^{-1}$ applied to the law in (40).

Having finished the conditioning calculations, we may now obtain the statistics of $\bm{A}$. Recall that $A_{ij}=\bm{v}_{i}^{\top}\bm{A}^{(2)}\bm{v}_{j}=\langle\frac{1}{2}(\bm{v}_{i}\bm{v}_{j}^{\top}+\bm{v}_{j}\bm{v}_{i}^{\top}),\bm{A}^{(2)}\rangle$. Applying $\mathsf{isovec}$ to each matrix and using the expression derived above, we find the mean and covariance

[TABLE]

Next, we make two simplifying approximations. For the means, we approximate

[TABLE]

which gives

[TABLE]

For the covariances, since under our assumptions we have $\|\mathsf{isovec}(\bm{v}_{i}\bm{v}_{i}^{\top})\|_{2}=\|\bm{v}_{i}\bm{v}_{i}^{\top}\|_{F}=\|\bm{v}_{i}\|_{2}^{2}=\delta$, we approximate

[TABLE]

which gives

[TABLE]

Finally, to recover what this prediction implies for the entries of $\bm{Z}^{\mathsf{mult}[2,2]}$, we compute

[TABLE]

We then choose $\sigma^{2}$ such that $Z_{(i,i)(i,i)}^{\mathsf{mult}[2,2]}=1$, which requires $\sigma^{2}=\delta^{-2}$, and, restricting to $i\neq j$ and $k\neq\ell$, we recover the same formula as (36):

[TABLE]

Remark 4.5**.**

It is worth noting the intriguing geometric interpretation of the random matrix $\bm{A}$ we have constructed: we have $\mathbb{E}\bm{A}=\bm{M}$, $\mathsf{diag}(\bm{A})=\bm{1}$ deterministically, and $\bm{A}$ fluctuates in the linear subspace $\mathsf{pert}_{\mathscr{E}_{2}^{N}}(\bm{M})$ (as may be verified from the covariance formula (42) and is intuitive by analogy with the ETF case of Section 4.1). Thus, $\bm{A}$ behaves, roughly speaking, like a random element of $\mathscr{E}_{2}^{N}$ (except that there is no enforcement of positive semidefiniteness), which lies on the same face of $\mathscr{E}_{2}^{N}$ as $\bm{M}$ and fluctuates gaussianly about $\bm{M}$ along this face.

4.3 Precise construction details

Having intuitively motivated the a priori unusual degree 4 pseudomoment formula given (identically) in (36) and (48) in the previous two sections, we now give the precise details of how this may be adjusted to produce an actually valid degree 4 pseudomoment extension of $(1-\alpha)\bm{M}+\alpha\bm{I}_{N}$, the “nudged” Montanari-Sen witness. It is instructive to view our construction as first attempting to build an extension of $\bm{M}$ itself, then introducing the adjustment towards $\bm{I}_{N}$ as a necessity to ensure positive semidefiniteness.

Step 1: Heuristic pseudomoments

We first build $\bm{X}\in\mathbb{R}^{\binom{[N]}{\leq 2}\times\binom{[N]}{\leq 2}}$ that is a reasonable prediction of a reduced degree 4 pseudomoment extension of $\bm{M}$. Viewing $\bm{X}$ as a $3\times 3$ block matrix as in (31), all blocks but the lower right are prescribed by the properties of being reduced and extending $\bm{M}$, so we have

[TABLE]

We complete the definition by defining $\bm{X}^{[2,2]}$ using the heuristics described earlier. Namely, we take

[TABLE]

Reviewing the constraints required of $\bm{X}$ per Proposition 1.11, we see that $\bm{X}$ satisfies the permutation symmetry constraints (Condition 5 in the Proposition) and the normalization and reduction constraints (Conditions 2 and 3, respectively) exactly, and satisfies the other linear constraints which require $X^{[2,2]}_{\{i,j\}\{i,k\}}=M_{jk}$ (Condition 4) approximately. Finally, the discussion of Sections 4.1 and 4.2 suggests that $\bm{X}$ should also be positive semidefinite (Condition 1).

Step 2: Correction to satisfy linear constraints

We next correct $\bm{X}$ to satisfy exactly all linear constraints required for a degree 4 pseudomoment matrix, by adjusting $\bm{X}^{[2,2]}$ to satisfy Condition 4 of Proposition 1.11. Define an additive correction $\bm{\Delta}\in\mathbb{R}^{\binom{[N]}{2}\times\binom{[N]}{2}}$ by

[TABLE]

(Note that the second part of the definition is consistent when $k=\ell$ regardless of whether we view $i$ or $k$ as the repeated index.)

Then, we set

[TABLE]

and $\bm{Y}$ satisfies Conditions 2 through 5 of Proposition 1.11 exactly. However, as we will see, $\bm{X}^{[2,2]}$ is a low-rank matrix, while $\bm{\Delta}$ acts non-trivially on components of the null space of $\bm{X}^{[2,2]}$. Therefore, even if $\bm{X}\succeq\bm{0}$ (which, as we will see, is nearly true), we will still have $\bm{Y}\not\succeq\bm{0}$ due to the fluctuations in $\bm{\Delta}$.

Step 3: Correction to satisfy positive semidefiniteness

Finally, we introduce a second correction to counteract the fluctuations in the spectra of $\bm{X}^{[2,2]}$ and $\bm{\Delta}$. We note that the identity matrix $\bm{I}_{\binom{[N]}{\leq 2}}$ is in fact a valid degree 4 pseudomoment matrix, which extends $\bm{I}_{N}\in\mathscr{C}^{N}$. Indeed, $\bm{I}_{N}$ is a natural choice of a point of $\mathscr{C}^{N}$ towards which to “push” $\bm{M}$ in order to regularize our construction, because $\bm{I}_{N}$ is the barycenter of the vertices of $\mathscr{C}^{N}$, i.e., $\bm{I}_{N}=\frac{1}{2^{N}}\sum_{\bm{x}\in\{\pm 1\}^{N}}\bm{x}\bm{x}^{\top}$.

Following this intuition, given a choice of the parameter $\alpha\in(0,1)$, we set

[TABLE]

Clearly, $\bm{Z}$ extends $(1-\alpha)\bm{M}+\alpha\bm{I}_{N}\in\mathscr{E}_{2}^{N}$ and satisfies all linear constraints on a degree 4 pseudomoment matrix (since both $\bm{Y}$ and $\bm{I}_{\binom{[N]}{\leq 2}}$ do so). Thus to show $(1-\alpha)\bm{M}+\alpha\bm{I}_{N}\in\mathscr{E}_{4}^{N}$, it suffices to show $\bm{Z}\succeq\bm{0}$, in which case $\bm{Z}$ will be a degree 4 pseudomoment extension. Theorem 2.1 will then be proved if we show that $\bm{Z}\succeq\bm{0}$ with high probability.

5 Conjectural higher-degree extension

Before proceeding to the proofs, we mention that there is a natural extension of the heuristic for pseudomoment construction of Section 4.2 that appears promising, though difficult to analyze, for higher-degree SOS relaxations.

The idea is to view higher-order pseudomoments as the second moments of symmetric gaussian tensors, which, as done in Section 4.2 for matrices, are formed by conditioning a certain canonical symmetric tensor distribution on desirable properties. Suppose we want to predict the degree $d=2k$ pseudomoment extension $\bm{Z}^{\mathsf{mult}[k,k]}\in\mathbb{R}^{[N]^{k}\times[N]^{k}}$ of the Montanari-Sen witness $\bm{M}\in\mathscr{E}_{2}^{N}$ (which, as before, we assume to be an exact unit norm tight frame, i.e., a constant multiple of $\bm{P}$), where, extending the case $k=2$ from Section 4.2, the entry indexed by $\bm{i},\bm{j}\in[N]^{k}$ is the pseudoexpectation of $\prod_{\ell=1}^{k}x_{i_{\ell}}\prod_{\ell=1}^{k}x_{j_{\ell}}$. We do this by building a symmetric tensor $\bm{A}^{(k)}\in\mathsf{Sym}^{k}(\mathbb{R}^{N})$ with jointly gaussian entries, and setting, for $\bm{i},\bm{j}\in[N]^{k}$,

[TABLE]

To describe the law of $\bm{A}^{(k)}$, we first define the following tensorial generalization of the GOE (see, e.g., [RM14] for properties of this distribution analogous to those of the GOE).

Definition 5.1**.**

Let $\bm{G}\in(\mathbb{R}^{N})^{\otimes k}$ have i.i.d. entries distributed as $\mathcal{N}(0,1)$. Then, write $\mathcal{G}^{N,k}(\sigma^{2})$ for the law of $\bm{A}\in\mathsf{Sym}^{k}(\mathbb{R}^{N})\subset(\mathbb{R}^{N})^{\otimes k}$ defined by

[TABLE]

Now, we define $\bm{A}^{(k)}$ inductively over $k$ as a family of coupled gaussian tensors, and ensure that the pseudomoment matrices thus formed are consistent with one another. Namely, we proceed as follows. Let $\bm{i}\circ\bm{j}$ denote the concatenation of finite strings in the alphabet $[N]$.

Let $A^{(0)}_{\varnothing}=1$. 2. 2.

For $k\geq 1$, let $\bm{A}^{(k)}$ have the law $\mathcal{G}^{N,k}(\sigma_{k}^{2})$, conditioned on the following two properties:

• •

(Subspace Property) For $\bm{i}\in[N]^{k-1}$, let $\bm{A}^{(k)}[\bm{i}]\colonequals(A_{\bm{i}\circ(j)}^{(k)})_{j=1}^{N}\in\mathbb{R}^{N}$. Then, for all $\bm{i}\in[N]^{k-1}$, $(\bm{I}-\bm{P})\bm{A}^{(k)}[\bm{i}]=\bm{0}$.

• •

(Consistency Property) For $\bm{i}\in[N]^{k-2}$ and $j\in[N]$, $A^{(k)}_{\bm{i}\circ(jj)}=A^{(k-2)}_{\bm{i}}$.

The constants $\sigma_{k}^{2}$ remain as free parameters to be tuned to ensure normalization, as in the case $k=2$ from Section 4.2.

Based on this reasonable generalization, we offer two conjectures. First, we believe that whatever adjustments are necessary to this construction are already captured in the simple adjustment of the Montanari-Sen witness towards the identity matrix given in Theorem 2.1.

Conjecture 5.2**.**

For any $\alpha,\delta\in(0,1)$ and $d\in 2\mathbb{N}$,

[TABLE]

More specifically but less formally, we believe the construction presented above is approximately the correct degree $d$ pseudomoment extension.

Conjecture 5.3** (Informal).**

For $\delta\in(0,1)$, let $\bm{M}^{(\delta)}$ be the Montanari-Sen witness. Then, for fixed $d\in 2\mathbb{N}$ and constants $\sigma_{k}^{2}$ depending only on $\delta$, with high probability as $N\to\infty$, the entries of $\bm{Z}^{\mathsf{mult}[k,k]}$ as defined above give a “nearly” valid degree $d=2k$ pseudomoment extension of $\bm{M}^{(\delta)}$.

We have tested Conjecture 5.3 numerically on Laurent’s construction [Lau03] of higher-degree pseudomoment matrices extending a deterministic $\bm{M}\in\mathscr{E}_{2}^{N}$, which indeed forms the Gram matrix of an ETF. Laurent’s construction shows that certain parity inequalities holding over $\mathscr{C}^{N}$ are not certified by SOS until $d\sim N$. We find that the results, for suitable tuning of $\sigma_{k}^{2}$, agree with Laurent’s construction (with no further adjustment needed). Thus one pleasant consequence of verifying Conjecture 5.3 may be a novel proof of Laurent’s theorem, whose original proof involves first predicting the entries of the pseudomoment matrix and then appealing to a technical analysis of hypergeometric functions to verify positive semidefiniteness.

We remark that we have also verified algebraically in our earlier paper [BK18] that, for degree 4, the construction in Theorem 4.4 (before simplifying in the $N\to\infty$ asymptotic regime) exactly recovers Laurent’s construction. This also would not be the first simplified proof of Laurent’s theorem; for instance, the work [KLM16] gives a general treatment of sum-of-squares relaxations of combinatorial optimization problems with highly symmetric formulations, and also produces a Cholesky-type decomposition of the relevant pseudomoment matrix. However, the idea we outline above both would unify such results with those for less symmetric problems, and would give a natural interpretation of the Cholesky factors in such a decomposition as the random variables that are the entries of the tensors $\bm{A}^{(k)}$.

Finally, let us remark on what seems to be the major difficulty in analyzing this construction. By analogy with the analysis in Section 4.2, we are eventually led, in conditioning on the Consistency Property, to attempt to approximate the orthogonal projector to the “repeated indices subspace”

[TABLE]

(Here $\odot$ denotes the symmetric product of tensors; see, e.g., [SKM89] for definitions. We mean “orthogonal projection” with respect to the Frobenius or entrywise inner product of general non-symmetric tensors, into which symmetric tensors are embedded by repeating entries.) When $k=2$, the spanning set consists of the $N$ linearly independent and roughly orthogonal tensors $\bm{v}_{i}^{\odot 2}$, which allows the orthogonal projection to be estimated by the sum of rank one projections as in (45). However, when $k\geq 3$, there does not appear to be a clear way to choose a convenient approximately-orthogonal basis to carry out the calculation. The collection of tensors $\bm{v}_{i}^{\odot 2}\odot\bm{v}_{j_{1}}\odot\cdots\odot\bm{v}_{j_{k-2}}$ is highly overcomplete, since the $\bm{v}_{i}$ themselves are an overcomplete set in $\mathbb{R}^{r}$ (and the symmetric tensor product is distributive, so any dependence among the $\bm{v}_{i}$ is inherited by the $\bm{A}\odot\bm{v}_{i}$ for any symmetric tensor $\bm{A}$). Moreover, even the subspaces $V^{(k)}_{i}\colonequals\bm{v}_{i}^{\odot 2}\odot\mathsf{Sym}^{k-2}(\mathbb{R}^{r})$ intersect non-trivially; for instance, $\bm{v}_{i}^{\odot 2}\odot\bm{v}_{j}^{\odot 2}\in V^{(4)}_{i}\cap V^{(4)}_{j}$. Thus it appears that a deeper understanding of the structure of these subspaces of symmetric tensors is required to form the correct higher-degree analogue of (45).

6 Proof of positive semidefiniteness: first steps

Recall that, in Section 4.3, we built from the Montanari-Sen witness $\bm{M}$ and an additional constant $\alpha\in(0,1)$ the matrix

[TABLE]

and found that to prove Theorem 2.1 it suffices to show that $\bm{Z}\succeq\bm{0}$ with high probability. We now give some technical preliminaries for the proof of this.

First, note that, after permuting rows and columns, $\bm{Z}$ is the direct sum of $(1-\alpha)\bm{M}+\alpha\bm{I}_{N}$, which is positive semidefinite by assumption, with the principal minor of $\bm{Z}$ indexed by $\{\varnothing\}\cup\binom{[N]}{2}$. Thus to show that $\bm{Z}\succeq\bm{0}$ it suffices to show that the latter minor is positive semidefinite.

Second, we may reduce the dimensionality of this remaining task by taking the Schur complement criterion for positive semidefiniteness with respect to the upper left entry, indexed by $(\varnothing,\varnothing)$, whose value is 1. The condition of positive semidefiniteness of the Schur complement is then

[TABLE]

We reorganize this expression as

[TABLE]

To show $\bm{Z}\succeq\bm{0}$, it then suffices to show that both $\bm{Z}^{(1)}\succeq\bm{0}$ and $\bm{Z}^{(2)}\succeq\bm{0}$. We refer to these as the “main term” and the “correction term,” respectively.

In this decomposition, we split between $\bm{Z}^{(1)}$ and $\bm{Z}^{(2)}$ the extra term $\alpha\bm{I}_{N(N-1)/2}$ that we introduced when nudging our pseudomoment matrix towards the identity. This term will act as a “barrier” against small fluctuations that might spoil positive semidefiniteness. We will show that without this adjustment $\bm{Z}^{(1)}$ and $\bm{Z}^{(2)}$ are nearly positive semidefinite already, having the magnitude of their smallest (most negative) eigenvalue tending to zero as $N\to\infty$ for any fixed $\delta$. Thus any choice of $\alpha>0$ will suffice to ensure that $\bm{Z}\succeq\bm{0}$ with high probability.

More specifically, we will show the following results.

Lemma 6.1** (Control of main term).**

For all $\delta\in(0,1)$,

[TABLE]

Lemma 6.2** (Control of correction term).**

For all $\delta\in(0,1)$,

[TABLE]

From Lemmata 6.1 and 6.2, it follows that with high probability $\bm{Z}\succeq\bm{0}$ for any $\alpha\in(0,1)$ fixed as $N\to\infty$ (or for $\alpha=\alpha(N)$ decreasing sufficiently slowly with $N$, though for the sake of simplicity we will not pursue this minor strengthening of the results). Theorem 2.1 then follows. It remains only to prove the Lemmata; in Section 7 we will prove Lemma 6.1, and in Section 8 we will prove Lemma 6.2.

7 Proof of positive semidefiniteness: main term

In this section we prove Lemma 6.1. We have

[TABLE]

Consider the quadratic form $\bm{a}^{\top}\bm{Z}^{(1a)}\bm{a}$, where we think of $a_{\{i,j\}}=2A_{ij}$ for some symmetric matrix $\bm{A}$ with $\mathsf{diag}(\bm{A})=\bm{0}$ (i.e., $\bm{a}=2\cdot\mathsf{offdiag}(\bm{A})$). Writing $\bm{m}_{1},\dots,\bm{m}_{N}\in\mathbb{R}^{N}$ for the columns of $\bm{M}$,

[TABLE]

Writing the above as a quadratic form in $\mathsf{isovec}(\widehat{\bm{V}}\bm{A}\widehat{\bm{V}}^{\top})$, we obtain the following.

Proposition 7.1**.**

Define

[TABLE]

(Recall that $\bm{1}_{\mathsf{diag}}\colonequals\mathsf{isovec}(\bm{I}_{r})$.) Then,

[TABLE]

Proof.

Since the right-hand side of (79) is at most zero, it suffices to consider the case that $\lambda_{\min}(\bm{Z}^{(1a)})<0$. By (77), we have

[TABLE]

Note first that

[TABLE]

Then, recalling that $\|\bm{a}\|_{2}^{2}=2\|\bm{A}\|_{F}^{2}$ and $\|\mathsf{isovec}(\widehat{\bm{V}}\bm{A}\widehat{\bm{V}}^{\top})\|_{2}^{2}=\|\widehat{\bm{V}}\bm{A}\widehat{\bm{V}}^{\top}\|_{F}^{2}$, by the variational description of the minimum eigenvalue we have

[TABLE]

completing the proof. ∎

We will thus focus our attention on $\widetilde{\bm{Z}}^{(1a)}$. Analyzing the Wishart-type matrix formed by the third term of (78), $\sum_{i=1}^{N}\mathsf{isovec}(\widehat{\bm{v}}_{i}\widehat{\bm{v}}_{i})\mathsf{isovec}(\widehat{\bm{v}}_{i}\widehat{\bm{v}}_{i})^{\top}$, will be our main difficulty. Since $\mathbb{E}\widehat{\bm{v}}_{i}\widehat{\bm{v}}_{i}^{\top}=\frac{1}{r}\bm{I}_{r}$, we center the vectors involved, and decompose this term as

[TABLE]

The remaining analysis involves a delicate balance between requirements in controlling $\bm{T}^{(1)}$ and $\bm{T}^{(2)}$. In order to bound the cross-term $\bm{T}^{(1)}$, we will rely on the strong concentration of the eigenvalues of $\widehat{\bm{V}}\widehat{\bm{V}}^{\top}$ that is created by the dependencies among the $\widehat{\bm{v}}_{i}$ (this is the “near-UNTF Gram matrix” behavior of $\bm{M}$). In particular, this concentration is much stronger than if $\widehat{\bm{v}}_{i}$ were replaced with any reasonable distribution of i.i.d. unit vectors, and this portion of our argument would fail for i.i.d. vectors (see Remark 7.3).

On the other hand, in order to bound the term $\bm{T}^{(2)}$, we will need to take advantage of the weak dependence of the $\widehat{\bm{v}}_{i}$, and formalize the intuition that because $N\ll r(r+1)/2$ and $\bm{T}^{(2)}$ is a sum of weakly dependent rank-one orthogonal projectors, $\bm{T}^{(2)}$ should itself behave approximately as an orthogonal projector to a subspace of dimension $N$ (though we will discuss one important caveat to this intuition in Remark 7.3). Technically, we will appeal to Lipschitz concentration inequalities for the Haar measure on Stiefel manifolds, which capture the heuristic weak dependence of entries of blocks of random orthogonal matrices under the Haar measure.

7.1 Bounding the cross-term $\bm{T}^{(1)}$

Lemma 7.2**.**

For all $K>0$,

[TABLE]

Proof.

Applying the matrix arithmetic-geometric mean inequality,

[TABLE]

Rewriting the norm appearing in the first term,

[TABLE]

since $\bm{V}\bm{V}^{\top}=\bm{I}_{r}$. By Proposition 3.1,

[TABLE]

Thus with at least the same probability we have

[TABLE]

and the result follows. ∎

Remark 7.3**.**

Let us contrast the result of this section with the same analysis for i.i.d. vectors. The marginal law of each $\widehat{\bm{v}}_{i}$ is uniform over $\SS^{r-1}$, so consider taking $\widetilde{\bm{v}}_{i}\sim\mathsf{Unif}(\SS^{r-1})$ independent. Then, we compute

[TABLE]

Thus the corresponding cross-term would have largest eigenvalue of order $\delta^{-1}$, which in particular would not decay with $N$.

Consequently, our previous intuition that we should obtain an approximate projector of rank $N$ from $\bm{T}^{(2)}=\sum_{i=1}^{N}\mathsf{isovec}(\widehat{\bm{v}}_{i}\widehat{\bm{v}}_{i}^{\top}-\frac{1}{r}\bm{I}_{r})\mathsf{isovec}(\widehat{\bm{v}}_{i}\widehat{\bm{v}}_{i}^{\top}-\frac{1}{r}\bm{I}_{r})^{\top}$ cannot be correct, since the putative basis vectors $\mathsf{isovec}(\widehat{\bm{v}}_{i}\widehat{\bm{v}}_{i}^{\top}-\frac{1}{r}\bm{I}_{r})$ almost sum to zero. In the following section, we will show that this is in fact the only linear near-dependence of these vectors, and $\bm{T}^{(2)}$ is still an approximate orthogonal projector, only of rank $N-1$.

7.2 Bounding the projection term $\bm{T}^{(2)}$: unnormalized case

Our strategy for bounding $\bm{T}^{(2)}$ will proceed in two steps: first, we will bound the same matrix but constructed from the approximately normalized vectors $\delta^{-1/2}\bm{v}_{1},\dots,\delta^{-1/2}\bm{v}_{N}$ in place of the strictly normalized vectors $\widehat{\bm{v}}_{1},\dots,\widehat{\bm{v}}_{N}$, and then we will show that this replacement does not significantly affect the spectrum. In this section we perform the first, more difficult of these tasks. We will show the following result.

Lemma 7.4**.**

Let $\bm{A}^{{(\mathsf{orth})}}$ have $\mathsf{isovec}(\delta^{-1}\bm{v}_{i}\bm{v}_{i}^{\top}-\frac{1}{r}\bm{I}_{r})$ as its columns. Let $\bm{P}_{\bm{1}_{N}^{\top}}\colonequals\bm{I}_{N}-\frac{1}{N}\bm{1}_{N}\bm{1}_{N}^{\top}$, the orthogonal projector to the subspace orthgonal to $\bm{1}_{N}$. Then,

[TABLE]

The argument will use the technique of union bounding over a net. Our main technical tool will be the following Lipschitz concentration inequality for the Haar measure of the Stiefel manifolds.

Recall that the Stiefel manifolds are defined as:

[TABLE]

The Haar measure $\mathsf{Haar}(\mathsf{Stief}(N,r))$ may be viewed as the measure obtained by restricting $\mathsf{Haar}(\mathcal{O}(N))$ to the upper $r\times N$ matrix block.

These measures enjoy the following concentration inequality when $r<N$, obtained by standard arguments from logarithmic Sobolev or isoperimetric inequalities for the special orthogonal group $\mathcal{SO}(N)$, of which $\mathsf{Stief}(N,r)$ is a quotient when $r<N$ (see, e.g., the discussion following Theorem 2.4 of [Led01]).

Proposition 7.5**.**

Suppose $1\leq r<N$, and $F:\mathsf{Stief}(N,r)\to\mathbb{R}$ has Lipschitz constant at most $L$ when $\mathsf{Stief}(N,r)$ is endowed with the metric of the Frobenius matrix norm. Then, for an absolute constant $C>0$,

[TABLE]

Note that since we have $r=\delta N$ with $\delta<1$, we will always satisfy the hypothesis $r<N$; we will use this implicitly without further comment for the remainder of the proof.

Proof of Lemma 7.4.

Note that $\bm{A}^{{(\mathsf{orth})}}\bm{1}_{N}=\mathsf{isovec}(\frac{N}{r}\bm{V}\bm{V}^{\top}-\frac{N}{r}\bm{I}_{r})=\bm{0}$; thus as suggested already in Remark 7.3, it is impossible for $\bm{A}^{{(\mathsf{orth})}}$ to act on $\mathbb{R}^{N}$ as an approximate isometric embedding, as we might naively expect from its weakly dependent columns. Our argument is more natural to carry out if we remove this caveat; therefore, let us define $\bm{A}^{(\mathsf{orth})}_{0}$ to have columns $\mathsf{isovec}(\frac{N}{r}\bm{v}_{i}\bm{v}_{i}^{\top}-\frac{1-\sqrt{\delta}}{r}\bm{I}_{r})$. One may check that $\|\bm{A}^{(\mathsf{orth})}_{0}\bm{1}_{N}\|_{2}=\|\bm{1}_{N}\|_{2}=\sqrt{N}$, and that

[TABLE]

In particular, $\bm{A}_{0}^{{(\mathsf{orth})}^{\top}}\bm{A}_{0}^{(\mathsf{orth})}-\bm{I}_{N}=\bm{A}^{{(\mathsf{orth})}^{\top}}\bm{A}^{(\mathsf{orth})}-\bm{P}_{\bm{1}_{N}^{\perp}}$, so it suffices to show the operator norm bound of (92) for $\bm{A}_{0}^{{(\mathsf{orth})}^{\top}}\bm{A}_{0}^{(\mathsf{orth})}-\bm{I}_{N}$.

For $\bm{x}\in\mathbb{R}^{N}$, let us denote $\bm{D}_{\bm{x}}\colonequals\mathsf{diag}(\bm{x})$ for the course of this proof. Then,

[TABLE]

For $\bm{x},\bm{y}\in\mathbb{R}^{N}$, define

[TABLE]

Then, recalling that $\bm{P}\colonequals\bm{V}^{\top}\bm{V}$ is the orthogonal projector to the row space of $\bm{V}$,

[TABLE]

Let us denote balls in Euclidean space by

[TABLE]

Our first goal will be to obtain concentration bounds on $F_{\bm{x},\bm{y}}(\bm{V})$ when $\bm{V}\sim\mathsf{Haar}(\mathsf{Stief}(N,r))$ for each fixed pair $(\bm{x},\bm{y})\in B(\bm{0},1)^{2}$, by applying the Lipschitz concentration inequality.

Claim 7.6**.**

Let $\bm{x},\bm{y}\in B(\bm{0},1)$. Then,

[TABLE]

Proof.

For $\bm{V}_{1},\bm{V}_{2}\in\mathsf{Stief}(N,r)$, letting $\bm{P}_{i}=\bm{V}_{i}^{\top}\bm{V}_{i}$, we have using (96) and the triangle inequality

[TABLE]

Since $\bm{P}_{i}$ is an orthogonal projector for $i\in\{1,2\}$,

[TABLE]

We bound the other term by

[TABLE]

Combining these observations and a symmetric argument with $\bm{x}$ and $\bm{y}$ in opposite roles gives

[TABLE]

Lastly, we bound

[TABLE]

where we have used that $\bm{V}_{1}\bm{V}_{1}^{\top}=\bm{V}_{2}\bm{V}_{2}^{\top}=\bm{I}_{r}$, and the result follows. ∎

Therefore, and what is crucial to our argument, while for the worst-case $\bm{x}\in B(\bm{0},1)$, namely $\bm{x}=\bm{e}_{i}$ a standard basis vector, $F_{\bm{x},\bm{x}}$ will have Lipschitz constant $O(1)$, for typical $\bm{x}\in B(\bm{0},1)$, $F_{\bm{x},\bm{x}}$ will rather have Lipschitz constant $\widetilde{O}(N^{-1/2})$. Moreover, the Lipschitz constant of $F_{\bm{x},\bm{y}}$ is comparable to the smaller of the Lipschitz constants of $F_{\bm{x},\bm{x}}$ and $F_{\bm{y},\bm{y}}$.

Claim 7.7**.**

For $\bm{x},\bm{y}\in B(\bm{0},1)$, $\mathbb{E}_{\bm{V}\sim\mathsf{Haar}(\mathsf{Stief}(N,r))}F_{\bm{x},\bm{y}}(\bm{V})=\langle\bm{x},\bm{y}\rangle+O_{\delta}(N^{-1})$.

Proof.

We have

[TABLE]

View $\bm{V}\sim\mathsf{Haar}(\mathsf{Stief}(N,r))$ as the top $r\times N$ block of $\bm{Q}\sim\mathsf{Haar}(\mathcal{O}(N))$. Then, expanding the first term with the moment formulae of Proposition 3.4,

[TABLE]

and since $|\langle\bm{x},\bm{y}\rangle|\leq\|\bm{x}\|_{2}\|\bm{y}\|_{2}\leq 1$ and $|\langle\bm{1}_{N},\bm{x}\rangle|\cdot|\langle\bm{1}_{N},\bm{y}\rangle|\leq N$, the result follows. ∎

Combining Claim 7.6, Claim 7.7, and the concentration result Proposition 7.5, we find the following corollary on pointwise concentration of $F_{\bm{x},\bm{y}}(\bm{V})$.

Claim 7.8**.**

There exist constants $C_{1},C_{2}>0$ depending only on $\delta$ such that, for any $\bm{x},\bm{y}\in B(\bm{0},1)$,

[TABLE]

This concludes the first part of the argument.

The remaining part of the argument is to apply a union bound of the probabilities controlled in Claim 7.8 over suitable nets of $B(\bm{0},1)$. We divide our task into a bound over sparse vectors and vectors with bounded largest entry, very similar to the technique in [Rud08, RV08] and especially [Ver11]. Introduce a parameter $\rho\in(0,1)$ to be chosen later. Define

[TABLE]

For any $\bm{x}\in B(\bm{0},1)$, we define $\bm{y}=\bm{y}(\bm{x})$ and $\bm{z}=\bm{z}(\bm{x})$ by thresholding the entries of $\bm{x}$, setting $y_{i}\colonequals x_{i}\mathbbold{1}\{|x_{i}|>\frac{1}{\sqrt{\rho N}}\}$ and $z_{i}\colonequals x_{i}\mathbbold{1}\{|x_{i}|\leq\frac{1}{\sqrt{\rho N}}\}$. Then, $\bm{x}=\bm{y}+\bm{z}$, $\bm{y}\in B_{s}$, and $\bm{z}\in B_{b}$.

Introduce another parameter $\gamma\in(0,1)$ to be chosen later. Let $N_{s}\subset B_{s}$ and $N_{b}\subset B_{b}$ be $\gamma$-nets. By a standard bound (see, e.g., Lemma 9.5 of [LT13]), we may choose $|N_{b}|\leq\exp(2N/\gamma)$, and by the same bound applied to each choice of $\rho N$ support coordinates for an element of $B_{s}$, we may choose

[TABLE]

To lighten the notation, let us set $\bm{S}\colonequals\bm{A}_{0}^{{(\mathsf{orth})}^{\top}}\bm{A}_{0}^{(\mathsf{orth})}-\bm{I}_{N}$. The following is an adaptation to our setting of a standard technique for estimating a matrix norm over a net: we first bound

[TABLE]

Rearranging this, we obtain

[TABLE]

Using Claim 7.8 and a union bound, we have that

[TABLE]

Taking $\rho=N^{-1/2}$, $t=C_{3}N^{-1/4}\log N$ for a large constant $C_{3}$, and $\gamma<\frac{1}{12}$ a small constant, we obtain the result. ∎

7.3 Bounding the projection term $\bm{T}^{(2)}$: normalization

In this section, we show that the passing from the approximately normalized vectors $\delta^{-1/2}\bm{v}_{i}$ discussed in the previous section to the exactly normalized vectors $\widehat{\bm{v}}_{i}=\bm{v}_{i}/\|\bm{v}_{i}\|_{2}$ does not affect the construction of $\bm{T}^{(2)}$ very much, as measured by operator norm.

Lemma 7.9**.**

Let $\bm{A}^{{(\mathsf{orth})}}\in\mathbb{R}^{r(r+1)/2\times N}$ have $\mathsf{isovec}(\delta^{-1}\bm{v}_{i}\bm{v}_{i}^{\top}-\frac{1}{r}\bm{I}_{r})$ as columns, and let $\bm{A}^{{(\mathsf{norm})}}\in\mathbb{R}^{r(r+1)/2\times N}$ have $\mathsf{isovec}(\widehat{\bm{v}}_{i}\widehat{\bm{v}}_{i}^{\top}-\frac{1}{r}\bm{I}_{r})$ as columns. Then, for all $K>0$,

[TABLE]

Proof.

Recall that $\bm{D}\in\mathbb{R}^{N\times N}$ is diagonal with $D_{ii}=\|\bm{v}_{i}\|_{2}^{2}$. Then, we may write

[TABLE]

Therefore,

[TABLE]

By Lemma 7.4 from the previous section, the second term is $O_{\delta}(1)$ with super-polynomially high probability. The result then follows by Proposition 3.1. ∎

Next, we translate this result to the Gram matrix $\bm{A}^{{(\mathsf{norm})}^{\top}}\bm{A}^{(\mathsf{norm})}$.

Corollary 7.10**.**

In the same setting as Lemma 7.9, for all $K>0$,

[TABLE]

Proof.

We may bound

[TABLE]

Then, combining Lemma 7.4 and Lemma 7.9 gives the result. ∎

7.4 Final main term bound: proof of Lemma 6.1

We are now ready to complete the proof of Lemma 6.1. First, we combine Lemma 7.4 and Corollary 7.10. Recall that these showed the following bounds, with high probability:

[TABLE]

Combining these, we obtain the following bound on $\bm{T}^{(2)}=\bm{A}^{(\mathsf{norm})}\bm{A}^{{(\mathsf{norm})}^{\top}}$.

Corollary 7.11**.**

For all $\delta\in(0,1)$,

[TABLE]

Next, we complete the proof of Lemma 6.1. Recall that

[TABLE]

and we have gathered the following bounds, holding with high probability:

[TABLE]

We can therefore control the minimum eigenvalue of $\widetilde{\bm{Z}}^{(1a)}$ by, with high probability,

[TABLE]

The first term is positive semidefinite for sufficiently large $N$ since $N/r^{2}\to 0$, and thus we find

[TABLE]

Finally, we must convert this to a bound on the smallest eigenvalue of $\bm{Z}^{(1a)}$. By Proposition 7.1, letting $\bm{P}=\bm{V}^{\top}\bm{V}$, we have

[TABLE]

By Proposition 3.1, with high probability $P_{ii}\geq\delta^{-1}-O_{\delta}(\sqrt{\frac{\log N}{N}})$ for all $i\in[N]$. Substituting this above, we thus find the result of Lemma 6.1,

[TABLE]

8 Proof of positive semidefiniteness: correction term

Proof of Lemma 6.2.

Recall that $\bm{\Delta}$ is non-zero only on pairs of index sets $\{i,j\}$ and $\{i,k\}$ that share an index. The non-zero entries are given by

[TABLE]

Let us compute the quadratic form of $\bm{\Delta}$ with $\bm{a}\in\mathbb{R}^{N(N-1)/2}$, which we view as having entries $a_{\{i,j\}}=A_{ij}$ for some $\bm{A}\in\mathbb{R}^{r\times r}_{\mathsf{sym}}$ with $\mathsf{diag}(\bm{A})=\bm{0}$ (i.e. $\bm{a}=\mathsf{offdiag}(\bm{A})$). We then have, expanding with a correction for double-counting the diagonal terms,

[TABLE]

We will now use the following inequality that allows us to bound inner products with the Schur square of a matrix by its Frobenius norm:

[TABLE]

(We denote by $\|\bm{Y}\|_{\ell^{\infty}}$ the vectorized supremum norm.) This will result in one term involving $\|\bm{A}\bm{M}\|_{F}^{2}$, which we bound by

[TABLE]

Combining these inequalities and noting that $\|\bm{a}\|_{2}^{2}=\frac{1}{2}\|\bm{A}\|_{F}^{2}$, we find (for $\bm{a}\neq\bm{0}$) that

[TABLE]

Since $\mathsf{diag}(\bm{M})=\bm{1}$, we have by Corollary 3.2 that, with high probability,

[TABLE]

and by Corollary 3.3, $\|\bm{M}\|_{\mathsf{op}}=O_{\delta}(1)$ with high probability.

To control the terms involving $\bm{R}$, recall that from Lemmata 7.4 and 7.9 it follows that with high probability

[TABLE]

Thus we have, with high probability,

[TABLE]

Combining these results, Lemma 6.2 follows. ∎

Remark 8.1**.**

We outline a simpler argument for Lemma 6.2 which suggests a sharper estimate, though it seems more difficult to formalize due to the dependency structure of $\bm{M}$. Since $\bm{\Delta}$ is sparse (with non-zero entries only when the row and column index sets $\{i,j\}$ and $\{k,\ell\}$ share an element), our strategy will be to apply the Gershgorin circle theorem. Thus we must bound the diagonal and off-diagonal entries of $\bm{\Delta}$.

The summation defining the diagonal entries contains only positive summands, so we have

[TABLE]

using Proposition 3.1 to control each term of the sum. (We omit bounds expressing “with high probability” statements and indulge in the logarithm-concealing $\widetilde{O}$ notation in this informal discussion.)

The off-diagonal entries of $\bm{\Delta}$ are more difficult to control. They are

[TABLE]

Here, we must capture the cancellations due to random signs: a naive application of the triangle inequality would give $|\Delta_{\{i,j\}\{i,k\}}|=\widetilde{O}(N^{-1})$, which would be insufficient for the Gershgorin circle theorem argument since there are $\Omega(N)$ non-zero entries in each row of $\bm{\Delta}$. On the other hand, a scaling like that in the central limit theorem with independent signs would give $|\Delta_{\{i,j\}\{i,k\}}|=\widetilde{O}(N^{-3/2})$, which would suffice, and would give $\|\bm{\Delta}\|_{\mathsf{op}}=\widetilde{O}(N^{-1/2})$, stronger by a factor of $N^{1/4}$ than the result of Lemma 6.2 (up to logarithmic factors).

Acknowledgements

We thank Jess Banks, Ankur Moitra, Andrea Montanari, Cristopher Moore, Tselil Schramm, and Ramon van Handel for useful discussions. We also thank the authors of [MRX19] for generously providing an early version of their manuscript.