On the Optimization Landscape of Tensor Decompositions

TL;DR

This paper analyzes the optimization landscape of over-complete tensor decomposition, showing that near-global maxima are prevalent among local optima, which explains the success of gradient ascent in solving these NP-hard problems.

Contribution

It provides the first analysis using Kac-Rice formula on the local minima of structured random polynomials in tensor decomposition, establishing conditions for near-global optimality.

Findings

All local maxima with function value close to the expectation are approximate global maxima.

Gradient ascent can reliably solve the tensor decomposition problem from barely better than random initialization.

First application of Kac-Rice formula to structured random polynomials with dependent coefficients.

Abstract

Non-convex optimization with local search heuristics has been widely used in machine learning, achieving many state-of-art results. It becomes increasingly important to understand why they can work for these NP-hard problems on typical data. The landscape of many objective functions in learning has been conjectured to have the geometric property that "all local optima are (approximately) global optima", and thus they can be solved efficiently by local search algorithms. However, establishing such property can be very difficult. In this paper, we analyze the optimization landscape of the random over-complete tensor decomposition problem, which has many applications in unsupervised learning, especially in learning latent variable models. In practice, it can be efficiently solved by gradient ascent on a non-convex objective. We show that for any small constant $\epsilon > 0$, among the set of points with function values $(1+\epsilon)$-factor larger than the expectation of the function, all the local maxima are approximate global maxima. Previously, the best-known result only characterizes the geometry in small neighborhoods around the true components. Our result implies that even with an initialization that is barely better than the random guess, the gradient ascent algorithm is guaranteed to solve this problem. Our main technique uses Kac-Rice formula and random matrix theory. To our best knowledge, this is the first time when Kac-Rice formula is successfully applied to counting the number of local minima of a highly-structured random polynomial with dependent coefficients.

Equations

$$T=\sum _{i=1}^n{a}_i\otimes a_i\otimes a_i\otimes a_i.$$

$$\max$$

$$s.t.$$

$$L=\left\{x\in S^{d-1}:f(x)\ge 3(1+\zeta )n\right\},$$

$$\text{grad }f(x)$$

$$\text{Hess }f(x)$$

$${\mathcal{M}}_f=\left\{x\in S^{d-1}:\text{grad }f(x)=0,\text{Hess }f(x)⪯0\right\}.$$

$$\mathbb{E}\text{displaylimits}\left[{\mathcal{M}}_f\cap Z|\right]={\int }_x\mathbb{E}\text{displaylimits}\left[|\det (\text{Hess }f)|\cdot 1(\text{Hess }f⪯0)1(x\in Z)\mid \text{grad }f(x)=0\right]p_{\text{grad }f(x)}(0)dx.$$

$$M$$

$$\mathbb{E}\text{displaylimits}\left[|{\mathcal{M}}_f\cap L|\right]=\text{Vol}(S^{d-1})\cdot \mathbb{E}\text{displaylimits}\left[\left|\det (M)\right|1(M⪰0)1({\Vert \alpha \Vert }_4^4\ge 3(1+\zeta )n)\mid g=0\right]p_g(0).$$

$$L_1:=\left\{x\in S^{d-1}:\sum _{i=1}^n⟨a_i,x{⟩}^4\ge 3n+\gamma \sqrt{nd}\right\}.$$

$$|{\mathcal{M}}_f\cap L_1|=2n.$$

$$L_1$$

$$\text{ where }{L}_2:=\{x\in S^{d-1}:\forall i,\Vert P_x{a}_i{\Vert }^2\ge (1-\delta )d,\text{and}|⟨a_i,x⟩{|}^2\le \delta d\}.$$

$$G_0:=$$

$$\forall U\subset [n]\text{ with }|U|<\delta n/\log d,\sum _{i\in U}⟨a_i,x{⟩}^2\le (1+\delta )d,$$

$$\sum _{i\in [n]}⟨a_i,x{⟩}^6\ge 15(1-\delta )n,$$

$$\displaystyle~{}~{}~{}n-3\sqrt{nd}\leq\sum_{i\in[n]}\langle a_{i},x\rangle^{2}\leq n+3\sqrt{nd}\Big{\}}\,.$$

$$\mathbb{E}\text{displaylimits}\left[|{\mathcal{M}}_f\cap L_1\cap L_2|\cdot 1(G_0)\right]\le 2^{-d/2}.$$

$$|{\mathcal{M}}_f\cap L_1\cap L_2^c|=2n.$$

$$\mathrm{Pr}\left[G_0\right]\ge 1-d^{-10}.$$

$$|{\mathcal{M}}_f\cap L_1\cap L_2|\cdot 1(G_0)\le \left|{\mathcal{M}}_f\cap L_1\cap L_2\cap L_G\right|,$$

$$\text{ (Hess f,grad f,1(x∈Z)) has the same distribution as (M,g,1(E0)1(E1)1(E2)1(E2′))},$$

$$E_0=$$

$${\Vert \alpha \Vert }_4^4\ge 15(1-\delta )n,$$

$$\displaystyle~{}~{}~{}~{}n-3\sqrt{nd}\leq\|\alpha\|^{2}\leq n+3\sqrt{nd}\Big{\}}\,.$$

$$E_1=$$

$$E_2$$

$$E_2^{\prime }$$

$$\mathbb{E}\text{displaylimits}\left[\left|{\mathcal{M}}_f\cap L_1\cap L_2\cap L_G\right|\right]=\text{Vol}(S^{d-1})\cdot \mathbb{E}\text{displaylimits}\left[\left|\det (M)\right|1(M⪰0)1(E_0)1(E_1)1(E_2)1(E_2^{\prime })\mid g=0\right]p_g(0).$$

$$\mathbb{E}\text{displaylimits}\left[\left|\det (M)\right|1(M⪰0)1(E)\mid g=0\right]p_g(0)=\mathbb{E}\text{displaylimits}\left[\mathbb{E}\text{displaylimits}\left[\left|\det (M)\right|1(M⪰0)1(E)\mid g=0,\alpha \right]p_{g\mid \alpha }(0)\right].$$

$$h(\alpha )$$

$$\mathbb{E}\text{displaylimits}\left[|{\mathcal{M}}_f\cap L_1\cap L_2|\cdot 1(G_0)\right]\le \mathbb{E}\text{displaylimits}\left[\left|{\mathcal{M}}_f\cap L_1\cap L_2\cap L_G\right|\right]=\mathbb{E}\text{displaylimits}\left[h(\alpha )\right].$$

$$W(\alpha ):=\mathbb{E}\text{displaylimits}\left[\det (M)1(M⪰0)1(E_2^{\prime })\mid g=0,\alpha \right]1(E_0)1(E_1)1(E_2).$$

$$h(\alpha )=\text{Vol}(S^{d-1})W(\alpha )p_{g\mid \alpha }(0).$$

Full text

In this paper, we analyze the optimization landscape of the random over-complete tensor decomposition problem, which has many applications in unsupervised learning, especially in learning latent variable models. In practice, it can be efficiently solved by gradient ascent on a non-convex objective. We show that for any small constant $\varepsilon>0$, among the set of points with function values $(1+\varepsilon)$-factor larger than the expectation of the function, all the local maxima are approximate global maxima. Previously, the best-known result only characterizes the geometry in small neighborhoods around the true components. Our result implies that even with an initialization that is barely better than the random guess, the gradient ascent algorithm is guaranteed to solve this problem.

Our main technique uses Kac-Rice formula and random matrix theory. To our best knowledge, this is the first time when Kac-Rice formula is successfully applied to counting the number of local minima of a highly-structured random polynomial with dependent coefficients.

1 Introduction

Non-convex optimization is the dominating algorithmic technique behind many state-of-art results in machine learning, computer vision, natural language processing and reinforcement learning. Local search algorithms through stochastic gradient methods are simple, scalable and easy to implement. Surprisingly, they also return high-quality solutions for practical problems like training deep neural networks, which are NP-hard in the worst case. It has been conjectured [DPG*+*14, CHM*+*15] that on typical data, the landscape of the training objectives has the nice geometric property that all local minima are (approximate) global minima. Such property assures the local search algorithms to converge to global minima [GHJY15, LSJR16, NP06, SQW15]. However, establishing it for concrete problems can be challenging.

Despite recent progress on understanding the optimization landscape of various machine learning problems (see [GHJY15, BBV16, BNS16, Kaw16, GLM16, HM16, HMR16] and references therein), a comprehensive answer remains elusive. Moreover, all previous techniques fundamentally rely on the spectral structure of the problems. For example, in [GLM16] allows us to pin down the set of the critical points (points with vanishing gradients) as approximate eigenvectors of some matrix. Among these eigenvectors we can further identify all the local minima. The heavy dependency on linear algebraic structure limits the generalization to problems with non-linearity (like neural networks).

Towards developing techniques beyond linear algebra, in this work, we investigate the optimization landscape of tensor decomposition problems. This is a clean non-convex optimization problem whose optimization landscape cannot be analyzed by the previous approach. It also connects to the training of neural networks with many shared properties [NPOV15] . For example, in comparison with the matrix case where all the global optima reside on a (connected) Grassmannian manifold, for both tensors and neural networks all the global optima are isolated from each other.

Besides the technical motivations above, tensor decomposition itself is also the key algorithmic tool for learning many latent variable models, mixture of Gaussians, hidden Markov models, dictionary learning [Cha96, MR06, HKZ12a, AHK12, AFH*+*12, HK13], just to name a few. In practice, local search heuristics such as alternating least squares [CLA09], gradient descent and power method [KM11] are popular and successful.

Concretely, we consider decomposing a random 4-th order tensor $T$ of the rank $n$ of the following form,

[TABLE]

We are mainly interested in the over-complete regime where $n\gg d$. This setting is particularly challenging, but it is crucial for unsupervised learning applications where the hidden representations have higher dimension than the data [AGMM15, DLCC07]. Previous algorithmic results either require access to high order tensors [BCMV14, GVX13], or use complicated techniques such as FOOBI [DLCC07] or sum-of-squares relaxation [BKS15, GM15, HSSS16, MSS16].

In the worst case, most tensor problems are NP-hard [Hås90, HL13]. Therefore we work in the average case where vectors $a_{i}\in\mathbb{R}^{d}$ are assumed to be drawn i.i.d from Gaussian distribution $\mathcal{N}(0,I)$. We call $a_{i}$’s the components of the tensor. We are given the entries of tensor $T$ and our goal is to recover the components $a_{1},\dots,a_{n}$.

We will analyze the following popular non-convex objective,

[TABLE]

It is known that for $n\ll d^{2}$, the global maxima of $f$ is close to one of $\pm\frac{1}{\sqrt{d}}a_{1},\dots,\pm\frac{1}{\sqrt{d}}a_{n}$. Previously, Ge et al. [GHJY15] show that for the orthogonal case where $n\leq d$ and all the $a_{i}$’s are orthogonal, objective function $f(\cdot)$ have only $2n$ local maxima that are approximately $\pm\frac{1}{\sqrt{d}}a_{1},\dots,\pm\frac{1}{\sqrt{d}}a_{n}$. However, the technique heavily uses the orthogonality of the components and is not generalizable to over-complete case.

Empirically, projected gradient ascent and power methods find one of the components $a_{i}$’s even if $n$ is significantly larger than $d$. The local geometry for the over-complete case around the true components is known: in a small neighborhood of each of $\pm\frac{1}{\sqrt{d}}a_{i}$’s, there is a unique local maximum [AGJ15]. Algebraic geometry techniques [CS13, ASS15] can show that $f(\cdot)$ has an exponential number of other critical points, while these techniques seem difficult to extend to the characterization of local maxima. It remains a major open question whether there are any other spurious local maxima that gradient ascent can potentially converge to.

Main results.

We show that there are no spurious local maxima in a large superlevel set that contains all the points with function values slightly larger than that of the random initialization.

Theorem 1.1**.**

Let $\varepsilon,\zeta\in(0,1/3)$ be two arbitrary constants and $d$ be sufficiently large. Suppose $d^{1+\varepsilon}<n<d^{2-\varepsilon}$. Then, with high probability over the randomness of $a_{i}$’s, we have that in the superlevel set

[TABLE]

there are exactly $2n$ local maxima with function values $(1\pm o(1))d^{2}$, each of which is $\widetilde{O}(\sqrt{n/d^{3}})$-close to one of $\pm\frac{1}{\sqrt{d}}a_{1},\dots,\pm\frac{1}{\sqrt{d}}a_{n}$.

Previously, the best known result [AGJ15] only characterizes the geometry in small neighborhoods around the true components, that is, there exists one local minima in each of the small constant neighborhoods around each of the true components $a_{i}$’s. (It turns out in such neighborhoods, the objective function is actually convex.) We significantly enlarge this region to the superlevel set $L$, on which the function $f$ is not convex and has an exponential number of saddle points, but still doesn’t have any spurious local minima.

Note that a random initialization $z$ on the unit sphere has expected function value $\mathbb{E}[f(z)]=3n$. Therefore the superlevel set $L$ contains all points that have function values barely larger than that of the random guess. Hence, Theorem 1.1 implies that with a slightly better initialization than the random guess, gradient ascent and power method111Power method is exactly equivalent to gradient ascent with a properly chosen finite learning rate are guaranteed to find one of the components in polynomial time. (It is known that after finding one component, it can be peeled off from the tensor and the same algorithm can be repeated to find all other components.)

Corollary 1.2**.**

In the setting of Theorem 1.1, with high probability over the choice of $a_{i}$’s, we have that given any starting point $x^{0}$ that satisfies $f(x^{0})\geq 3(1+\zeta)n$, stochastic projected gradient descent222We note that by stochastic gradient descent we meant the algorithm that is analyzed in [GHJY15]. To get a global minimizer in polynomial time (polynomial in $\log(1/\varepsilon)$ to get $\varepsilon$ precision), one also needs to slightly modify stochastic gradient descent in the following way: one can run SGD until $1/d$ accuracy and then switch to gradient descent. Since the problem is locally strongly convex, the local convergence is linear. will find one of the $\pm\frac{1}{\sqrt{d}}a_{i}$’s up to $\widetilde{O}(\sqrt{n/d^{3}})$ Euclidean error in polynomial time.

We also strengthen Theorem 1.1 and Corollary 1.2 (see Theorem 3.1) slightly – the same conclusion still holds with $\zeta=O(\sqrt{d/n})$ that is smaller than a constant. Note that the expected value of a random initialization is $3n$ and we only require an initialization that is slightly better than random guess in function value. We also conjecture that from random initialization, it suffices to use constant number of projected gradient descent (with optimal step size) to achieve the function value $3(1+\zeta)n$ with $\zeta=O(\sqrt{d/n})$. This conjecture — an interesting question for future work — is based on the hypothesis that the first constant number of steps of gradient descent can make similar improvements as the first step does (which is equal to $c\sqrt{dn}$ for a universal constant $c$).

As a comparison, previous works such as [AGJ15] require an initialization with function value $\Theta(d^{2})\gg n$. Anandkumar et al. [AGJ16] analyze the dynamics of tensor power method with a delicate initialization that is independent with the randomness of the tensor. Thus it is not suitable for the situation where the initialization comes from the result of another algorithm, and it does not have a direct implication on the landscape of $f(\cdot)$.

We note that the local maximum of $f(\cdot)$ corresponds to the robust eigenvector of the tensor. Using this language, our theorem says that a robust eigenvector of an over-complete tensor with random components is either one of those true components or has a small correlation with the tensor in the sense that $\langle T,x^{\otimes 4}\rangle$ is small. This improves significantly upon the understanding of robust eigenvectors [ASS15] under an interesting random model.

Our techniques

The proof of Theorem 1.1 uses Kac-Rice formula (see, e.g., [AT09]), which is based on a counting argument. To build up the intuition, we tentatively view the unit sphere as a collection of discrete points, then for each point $x$ one can compute the probability (with respect to the randomness of the function) that $x$ is a local maximum. Adding up all these probabilities will give us the expected number of local maxima. In continuous space, such counting argument has to be more delicate since the local geometry needs to be taken into account. This is formalized by Kac-Rice formula (see Lemma 2.2).

However, Kac-Rice formula only gives a closed form expression that involves the integration of the expectation of some complicated random variable. It’s often very challenging to simplify the expression to obtain interpretable results. Before our work, Auffinger et al. [AAČ13, AA*+*13] have successfully applied Kac-Rice formula to characterize the landscape of polynomials with random Gaussian coefficients. The exact expectation of the number of local minima can be computed there, because the Hessian of a random polynomial is a Gaussian orthogonal ensemble, whose eigenvalue distribution is well-understood with closed form expression.

Our technical contribution here is successfully applying Kac-Rice formula to structured random non-convex functions where the formula cannot be exactly evaluated. The Hessian and gradients of $f(\cdot)$ have much more complicated distributions compared to the Gaussian orthogonal ensemble. As a result, the Kac-Rice formula is impossible to be evaluated exactly. We instead cut the space $\mathbb{R}^{d}$ into regions and use different techniques to estimate the number of local maxima. See a proof overview in Section 3. We believe our techniques can be extended to 3rd order tensors and can shed light on the analysis of other non-convex problems with structured randomness.

Organization

In Section 2 we introduce preliminaries regarding manifold optimization and Kac-Rice formula. We give a detailed explanation of our proof strategy in Section 3. We fill in the technical details in the later sections: in Section 4 we show that there is no local maximum that is uncorrelated with any of the true components. We compliment that by a local analysis in Section 5 that shows there are exactly $2n$ local optima around the true components.

2 Notations and Preliminaries

We use $\textup{Id}_{d}$ to denote the identity matrix of dimension $d\times d$, and for a subspace $K$, let $\textup{Id}_{K}$ denote the projection matrix to the subspace $K$. For unit vector $x$, let $P_{x}=\textup{Id}-xx^{\top}$ denote the projection matrix to the subspace orthogonal to $x$. Let $S^{d-1}$ be the $d-1$-dimensional sphere $S^{d-1}:=\{x\in\mathbb{R}^{d}:\|x\|^{2}=1\}$.

Let $u\odot v$ denote the Hadamard product between vectors $u$ and $v$. Let $u^{\odot s}$ denote $u\odot\dots\odot u$ where $u$ appears $k$ times. Let $A\otimes B$ denote the Kronecker product of $A$ and $B$. Let $\|\cdot\|$ denote the spectral norm of a matrix or the Euclidean norm of a vector. Let $\lVert\cdot\rVert_{F}$ denote the Frobenius norm of a matrix or a tensor.

We write $A\lesssim B$ if there exists a universal constant $C$ such that $A\leq CB$. We define $\gtrsim$ similarly. Unless explicitly stated otherwise, $O(\cdot)$-notation hides absolute multiplicative constants. Concretely, every occurrence of the notation $O(x)$ is a placeholder for some function $f(x)$ that satisfies $\forall x\in\mathbb{R},|f(x)|\leq C|x|$ for some absolute constant $C>0$.

2.1 Gradient, Hessian, and local maxima on manifold

We have a constrained optimization problem over the unit sphere $S^{d-1}$, which is a smooth manifold. Thus we define the local maxima with respect to the manifold. It’s known that projected gradient descent for $S^{d-1}$ behaves pretty much the same on the manifold as in the usual unconstrained setting [BAC16]. In Section A we give a brief introduction to manifold optimization, and the definition of gradient and Hessian. We refer the readers to the book [AMS07] for more backgrounds. Here we use $\textup{grad }f$ and $\textup{Hess }f$ to denote the gradient and the Hessian of $f$ on the manifold $S^{d-1}$. We compute them in the following claim.

Claim 2.1**.**

Let $f:S^{d-1}\rightarrow\mathbb{R}$ be $f(x):=\frac{1}{4}\sum_{i=1}^{n}\langle a_{i},x\rangle^{4}$. Then the gradient and Hessian of $f$ on the sphere can be written as,

[TABLE]

where $P_{x}=\textup{Id}_{d}-xx^{\top}$.

A local maximum of a function $f$ on the manifold $S^{d-1}$ satisfies $\textup{grad }f(x)=0$, and $\textup{Hess }f(x)\preceq 0$. Let $\mathcal{M}_{f}$ be the set of all local maxima,

[TABLE]

2.2 Kac-Rice formula

Kac-Rice formula is a general tool for computing the expected number of special points on a manifold. Suppose there are two random functions $P(\cdot):\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$ and $Q(\cdot):\mathbb{R}^{d}\rightarrow\mathbb{R}^{k}$, and an open set $\mathcal{B}$ in $\mathbb{R}^{k}$. The formula counts the expected number of point $x\in\mathbb{R}^{d}$ that satisfies both $P(x)=0$ and $Q(x)\in\mathcal{B}$.

Suppose we take $P=\nabla f$ and $Q=\nabla^{2}f$, and let $\mathcal{B}$ be the set of negative semidefinite matrices, then the set of points that satisfies $P(x)=0$ and $Q\in\mathcal{B}$ is the set of all local maxima $\mathcal{M}_{f}$. Moreover, for any set $Z\subset S^{d-1}$, we can also augment $Q$ by $Q=[\nabla^{2}f,x]$ and choose $\mathcal{B}=\{A:A\preceq 0\}\otimes Z$. With this choice of $P,Q$, Kac-Rice formula can count the number of local maxima inside the region $Z$. For simplicity, we will only introduce Kac-Rice formula for this setting. We refer the readers to [AT09, Chapter 11&12] for more backgrounds.

Lemma 2.2** (Informally stated).**

Let $f$ be a random function defined on the unit sphere $S^{d-1}$ and let $Z\subset S^{d-1}$. Under certain regularity conditions333We omit the long list of regularity conditions here for simplicity. See more details at  [AT09, Theorem 12.1.1] on $f$ and $Z$, we have

[TABLE]

where $dx$ is the usual surface measure on $S^{d-1}$ and $p_{\textup{grad }f(x)}(0)$ is the density of $\textup{grad }f(x)$ at 0.

2.3 Formula for the number of local maxima

In this subsection, we give a concrete formula for the number of local maxima of our objective function (1.1) inside the superlevel set $L$ (defined in equation (1.2)). Taking $Z=L$ in Lemma 2.2, it boils down to estimating the quantity on the right hand side of (2.2). We remark that for the particular function $f$ as defined in (1.1) and $Z=L$, the integrand in (2.2) doesn’t depend on the choice of $x$. This is because for any $x\in S^{d-1}$, $(\textup{Hess }f,\textup{grad }f,\mathbf{1}(x\in L))$ has the same joint distribution, as characterized below:

Lemma 2.3**.**

Let $f$ be the random function defined in (1.1). Let $\alpha_{1},\dots,\alpha_{n}\in\mathcal{N}(0,1)$, and $b_{1},\dots,b_{n}\sim\mathcal{N}(0,\textup{Id}_{d-1})$ be independent Gaussian random variables. Let

[TABLE]

Then, we have that for any $x\in S^{d-1}$, $(\textup{Hess }f,\textup{grad }f,f)$ has the same joint distribution as $(-M,g,\left\lVert\alpha\right\rVert_{4}^{4})$.

Proof.

We use Claim 2.1. We fix $x\in S^{d-1}$ and let $\alpha_{i}=\langle a_{i},x\rangle$ and $b_{i}=P_{x}a_{i}$. We have $\alpha_{i}$ and $b_{i}$ are independent, and $b_{i}$ is spherical Gaussian random vector in the tangent space at $x$ (which is isomorphic to $\mathbb{R}^{d-1}$). We can verify that $\textup{grad }f(x)=\sum_{i\in[n]}\alpha_{i}^{3}b_{i}=g$ and $\textup{Hess }f(x)=-M$, and this complete the proof. ∎

Using Lemma 2.2 (with $Z=L$) and Lemma 2.3, we derive the following formula for the expectation of our random variable $\mathop{\mathbb{E}}\displaylimits\left[|\mathcal{M}_{f}\cap L|\right]$. Later we will later use Lemma 2.2 slightly differently with another choice of $Z$.

Lemma 2.4**.**

Using the notation of Lemma 2.3, let $p_{g}(\cdot)$ denote the density of $g$. Then,

[TABLE]

3 Proof Overview

In this section, we give a high-level overview of the proof of the main Theorem. We will prove a slightly stronger version of Theorem 1.1.

Let $\gamma$ be a universal constant that is to be determined later. Define the set $L_{1}\subset S^{d-1}$ as,

[TABLE]

Indeed we see that $L$ (defined in (1.2)) is a subset of $L_{1}$ when $n\gg d$. We prove that in $L_{1}$ there are exactly $2n$ local maxima.

Theorem 3.1** (main).**

There exists universal constants $\gamma,\beta$ such that the following holds: suppose $d^{2}/\log^{O(1)}\geq n\geq\beta d\log^{2}d$ and $L_{1}$ be defined as in (3.1), then with high probability over the choice of $a_{1},\dots,a_{n}$, we have that the number of local maxima in $L_{1}$ is exactly $2n$:

[TABLE]

Moreover, each of the local maximum in $L_{1}$ is $\widetilde{O}(\sqrt{n/d^{3}})$-close to one of $\pm\frac{1}{\sqrt{d}}a_{1},\dots,\pm\frac{1}{\sqrt{d}}a_{n}$.

In order to count the number of local maxima in $L_{1}$, we use the Kac-Rice formula (Lemma 2.4). Recall that what Kac-Rice formula gives an expression that involves the complicated expectation $\mathop{\mathbb{E}}\displaylimits\left[\left|\det(M)\right|\mathbf{1}(M\succeq 0)\mathbf{1}(\left\lVert\alpha\right\rVert_{4}^{4}\geq 3(1+\zeta)n)\mid g=0\right]$. Here the difficulty is to deal with the determinant of a random matrix $M$ (defined in Lemma 2.3), whose eigenvalue distribution does not admit an analytical form. Moreover, due to the existence of the conditioning and the indicator functions, it’s almost impossible to compute the RHS of the Kac-Rice formula (equation (2.4)) exactly.

Local vs. global analysis

The key idea to proceed is to divide the superlevel set $L_{1}$ into two subsets

[TABLE]

Here $\delta$ is a sufficiently small universal constant that is to be chosen later. We also note that $L_{2}^{c}\subset L_{1}$ and hence $L_{1}=(L_{1}\cap L_{2})\cup L_{2}^{c}$.

Intuitively, the set $L_{1}\cap L_{2}$ contains those points that do not have large correlation with any of the $a_{i}$’s; the compliment $L_{2}^{c}$ is the union of the neighborhoods around each of the desired vector $\frac{1}{\sqrt{d}}a_{1},\dots,\frac{1}{\sqrt{d}}a_{n}$. We will refer to the first subset $L_{1}\cap L_{2}$ as the global region, and refer to the $L_{2}^{c}$ as the local region.

We will compute the number of local maxima in sets $L_{1}\cap L_{2}$ and $L^{c}_{2}$ separately using different techniques. We will show that with high probability $L_{1}\cap L_{2}$ contains zero local maxima using Kac-Rice formula (see Theorem 3.2). Then, we show that $L^{c}_{2}$ contains exactly $2n$ local maxima (see Theorem 3.3) using a different and more direct approach.

Global analysis.

The key benefit of have such division to local and global regions is that for the global region, we can avoid evaluating the value of the RHS of the Kac-Rice formula. Instead, we only need to have an estimate: Note that the number of local optima in $L_{1}\cap L_{2}$, namely $|\mathcal{M}_{f}\cap L_{1}\cap L_{2}|$, is an integer nonnegative random variable. Thus, if we can show its expectation $\mathop{\mathbb{E}}\displaylimits\left[|\mathcal{M}_{f}\cap L_{1}\cap L_{2}|\right]$ is much smaller than $1$, then Markov’s inequality implies that with high probability, the number of local maxima will be exactly zero. Concretely, we will use Lemma 2.2 with $Z=L_{1}\cap L_{2}$, and then estimate the resulting integral using various techniques in random matrix theory. It remains quite challenging even if we are only shooting for an estimate. see Theorem 3.2 for the exact statement and Section 3.1 for an overview of the analysis.

Local analysis.

In the local region $L_{2}^{c}$, that is, the neighborhoods of $a_{1},\dots,a_{n}$, we will show there are exactly $2n$ local maxima. As argued above, it’s almost impossible to get exact numbers out of the Kac-Rice formula since it’s often hard to compute the complicated integral. Moreover, Kac-Rice formula only gives the expected number but not high probability bounds. However, here the observation is that the local maxima (and critical points) in the local region are well-structured. Thus, instead, we show that in these local regions, the gradient and Hessian of a point $x$ are dominated by the terms corresponding to components $\{a_{i}\}$’s that are highly correlated with $x$. The number of such terms cannot be very large (by restricted isometry property, see Section B.5). As a result, we can characterize the possible local maxima explicitly, and eventually show there is exactly one local maximum in each of the local neighborhoods around $\{\pm\frac{1}{\sqrt{d}}a_{i}\}$’s.

Concentration properties of $a_{i}$’s.

Before stating the formal results regarding the local and global analysis, we have the following technical preparation. Since $a_{i}$’s are chosen at random, with small probability the tensor $T$ will behave very differently from the average instances. The optimization problem for such irregular tensors may have much more local maxima. To avoid these we will restrict our attention to the following event $G_{0}$, which occurs with high probability (over the randomness of $a_{i}$’s) :

[TABLE]

Here $\delta$ is a small enough universal constant. Event $G_{0}$ summarizes common properties of the vectors $a_{i}$’s that we frequently use in the technical sections. Equation (3.4) is the restricted isometry property (see Section B.5) that ensures the $a_{i}$’s are not adversarially correlated with each other. Equation (3.5) lowerbounds the high order moments of $a_{i}$’s. Equation (3.6) bounds the singular values of the matrix $\sum_{i=1}^{n}a_{i}a_{i}^{\top}$. These conditions will be useful in the later technical sections.

Next, we formalize the intuition above as the two theorems below. Theorem 3.2 states there is no local maximum in $L_{1}\cap L_{2}$, whereas Theorem 3.3 concludes that there are exactly $2n$ local maxima in $L_{1}\cap L_{2}^{c}$.

Theorem 3.2**.**

There exists universal small constant $\delta\in(0,1)$ and universal constants $\gamma,\beta$ such that for sets $L_{1},L_{2}$ defined in equation (3.3) and $n\geq\beta d\log^{2}d$, we have that the expected number of local maxima in $L_{1}\cap L_{2}$ is exponentially small:

[TABLE]

Theorem 3.3**.**

Suppose $1/\delta^{2}\cdot d\log d\leq n\leq d^{2}/\log^{O(1)}d$. Then, with high probability over the choice $a_{1},\dots,a_{n}$, we have,

[TABLE]

Moreover, each of the point in $L\cap L_{2}^{c}$ is $\widetilde{O}(\sqrt{n/d^{3}})$-close to one of $\pm\frac{1}{\sqrt{d}}a_{1},\dots,\pm\frac{1}{\sqrt{d}}a_{n}$.

Proof of the main theorem

The following Lemma shows that event $G_{0}$ that we conditioned on is indeed a high probability event. The proof follows from simple concentration inequalities and is deferred to Section C.

Lemma 3.4**.**

Suppose $1/\delta^{2}\cdot d\log^{2}d\leq n\leq d^{2}/\log^{O(1)}d$. Then,

[TABLE]

Combining Theorem 3.2, Theorem 3.3 and Lemma 3.4, we obtain Theorem 3.1 straightforwardly.

Proof of Theorem 3.1.

By Theorem 3.2 and Markov inequality, we obtain that $\Pr\left[|L\cap L_{1}\cap L_{2}|\mathbf{1}(G_{0})<1/2\right]\leq 2^{-d/2+1}$. Since $|L\cap L_{1}\cap L_{2}|\mathbf{1}(G_{0})$ is an integer value random variable, therefore, we get $\Pr\left[|L\cap L_{1}\cap L_{2}|\mathbf{1}(G_{0})=0\right]\leq 2^{-d/2+1}$. Thus, using Lemma 3.4, Theorem 3.3 and union bound, with high probability, we have that $G_{0}$, $|L\cap L_{1}\cap L_{2}|\mathbf{1}(G_{0})=0$ and equation (3.7) happen. Then, using Theorem 3.3 and union bound, we concluded that $|L\cap L_{1}|=|L\cap L_{1}\cap L_{2}^{c}|+|L\cap L_{1}\cap L_{2}|=2n$. ∎

In the next subsections we sketch the basic ideas behind the proof of of Theorem 3.2 and Theorem 3.3. Theorem 3.2 is the crux of this technical part of the paper.

3.1 Estimating the Kac-Rice formula for the global region

The general plan to prove Theorem 3.2 is to use random matrix theory to estimate the RHS of the Kac-Rice formula. We begin by applying Kac-Rice formula to our situation.

We first note that by definition of $G_{0}$,

[TABLE]

where $L_{G}=\{x\in S^{d-1}:x~{}\textup{satisfies equation ~{}\eqref{eqn:32},~{}\eqref{eqn:29} and~{}\eqref{eqn:101}}\}$. Indeed, when $G_{0}$ happens, then $L_{G}=S^{d-1}$, and therefore equation (3.8) holds.

Thus it suffice to control $\mathop{\mathbb{E}}\displaylimits\left[\left|\mathcal{M}_{f}\cap L_{1}\cap L_{2}\cap L_{G}\right|\right]$. We will use the Kac-Rice formula (Lemma 2.2) with the set $Z=\mathcal{M}_{f}\cap L_{1}\cap L_{2}\cap L_{G}$.

Applying Kac-Rice formula.

The first step to apply Kac-Rice formula is to characterize the joint distribution of the gradient and the Hessian. We use the notation of Lemma 2.3 for expressing the joint distribution of $(\textup{Hess }f,\textup{grad }f,\mathbf{1}(x\in Z))$. For any fix $x\in S^{d-1}$, let $\alpha_{i}=\langle a_{i},x\rangle$ and $b_{i}=P_{x}a_{i}$ (where $P_{x}=\textup{Id}-xx^{\top}$) and $M=\lVert\alpha\rVert_{4}^{4}\cdot\textup{Id}_{d-1}-3\sum_{i=1}^{n}\alpha_{i}^{2}b_{i}b_{i}^{\top}~{}~{}\textup{and}~{}~{}g=\sum_{i=1}^{n}\alpha_{i}^{3}b_{i}$ as defined in (2.3). In order to apply Kac-Rice formula, we’d like to compute the joint distribution of the gradient and the Hessian. We have that

[TABLE]

where $E_{0},E_{1},E_{2},E_{2}^{\prime}$ are defined as follows (though these details here will only be important later).

[TABLE]

We see that the equations above correspond to equation (3.4), (3.5) and (3.6) and event $\mathbf{1}(E_{0})$ corresponds to the event $x\in L_{G}$. Similarly, the following event $E_{1}$ corresponds to $x\in L_{1}$.

[TABLE]

Events $E_{2}$ and $E_{2}^{\prime}$ correspond to the events that $x\in L_{2}$. We separate them out to reflect that $E_{2}$ and $E_{2}^{\prime}$ depends the randomness of $\alpha_{i}$’s and $b_{i}$’s respectively.

[TABLE]

Using Kac-Rice formula (Lemma 2.2 with $Z=L_{1}\cap L_{2}\cap L_{G}$), we conclude that

[TABLE]

Next, towards proving Theorem 3.2 we will estimate the RHS of the equation (3.10) using various techniques.

Conditioning on $\alpha$.

We observe that the distributions of the gradient $g$ and Hessian $M$ are fairly complicated. In particular, we need to deal with the interactions of $\alpha_{i}$’s (the components along $x$) and $b_{i}$’s (the components in the orthogonal subspace of $x$). Therefore, we use the law of total expectation to first condition on $\alpha$ and take expectation over the randomness of $b_{i}$’s, and then take expectation over $\alpha_{i}$’s. Here below the inner expectation of RHS of (3.11) is with respect to the randomness of $b_{i}$’s and the outer one is with respect to $\alpha_{i}$’s.

Lemma 3.5**.**

Using the notation of Lemma 2.3, let $E$ denotes an event and let $p_{g\mid\alpha}$ denotes the density of $g\mid\alpha$. Then,

[TABLE]

For notional convenience we define $h(\cdot):\mathbb{R}^{n}\rightarrow\mathbb{R}$ as

[TABLE]

Using equation (3.8), the Kac-Rice formula (equation (3.10)), and law of total expectation (Lemma 3.5), we obtain straightforwardly the following Lemma which gives an explicit formula for the number of local maxima in $L_{1}\cap L_{2}$. We provide a rigorous proof in Section C that verifies the regularity condition of Kac-Rice formula.

Lemma 3.6**.**

Let $h(\cdot)$ be defined as above. In the setting of this section, we have

[TABLE]

We note that $p_{g\mid\alpha}(0)$ has an explicit expression since $g\mid\alpha$ is Gaussian. For the ease of exposition, we separate out the hard-to-estimate part from $h(\alpha)$, which we call $W(\alpha)$:

[TABLE]

Therefore by definition, we have that

[TABLE]

Now, since we have conditioned on $\alpha$, the distributions of the Hessian, namely $M\mid\alpha$, is a generalized Wishart matrix which is slightly easier than before. However there are still several challenges that we need to address in order to estimate $W(\alpha)$.

Question I: how to control $\det(M)\mathbf{1}(M\succeq 0)$?

Recall that $M=\left\lVert\alpha\right\rVert_{4}^{4}-3\sum\alpha_{i}^{2}b_{i}b_{i}^{\top}$, which is a generalized Wishart matrix whose eigenvalue distribution has no (known) analytical expression. The determinant itself by definition is a high-degree polynomial over the entries, and in our case, a complicated polynomial over the random variables $\alpha_{i}$’s and vectors $b_{i}$’s. We also need to properly exploit the presence of the indicator function $\mathbf{1}(M\succeq 0)$, since otherwise, the desired statement will not be true – the function $f$ has an exponential number of critical points.

Fortunately, in most of the cases, we can use the following simple claim that bounds the determinant from above by the trace. The inequality is close to being tight when all the eigenvalues of $M$ are similar to each other. More importantly, it uses naturally the indicator function $\mathbf{1}(M\succeq 0)$! Later we will see how to strengthen it when it’s far from tight.

Claim 3.7**.**

For any $p\times p$ symmetric matrix $V$, we have,

[TABLE]

The claim is a direct consequence of AM-GM inequality.

Proof.

We can assume $V$ is positive semidefinite since otherwise both sides of the inequality vanish. Then, suppose $\lambda_{1},\dots,\lambda_{p}\geq 0$ are the eigenvalues of $V$. We have $\det(V)=\lambda_{1}\dots\lambda_{p}$ and $|\textup{tr}(V)|=\textup{tr}(V)=\lambda_{1}+\dots+\lambda_{p}$. Then by AM-GM inequality we complete the proof. ∎

Applying the Lemma above with $V=M$, we have

[TABLE]

Here we dropped the indicators for events $E_{2}$ and $E_{2}^{\prime}$ since they are not important for the discussion below. It turns out that $|\textup{tr}(M)|$ is a random variable that concentrates very well, and thus we have $\mathop{\mathbb{E}}\displaylimits\left[|\textup{tr}(M)|^{d-1}\right]\approx|\mathop{\mathbb{E}}\displaylimits\left[\textup{tr}(M)\right]|^{d-1}$. It can be shown that (see Proposition 4.3 for the detailed calculation),

[TABLE]

Therefore using equation (3.14) and equation above, ignoring $\mathbf{1}(E_{2}^{\prime})$, we have that

[TABLE]

Note that since $g\mid\alpha$ has Gaussian distribution, we have,

[TABLE]

Thus using two equations above, we can bound $\mathop{\mathbb{E}}\displaylimits\left[h(\alpha)\right]$ by

[TABLE]

Therefore, it suffices to control the RHS of (3.16), which is much easier than the original Kac-Rice formula. However, we still need to be careful here, as it turns out that RHS of (3.16) is roughly $c^{d}$ for some constant $c>1$! Roughly speaking, this is because the high powers of a random variables is very sensitive to its tail.

Easy case when all $\alpha_{i}$’s are small.

To find a tight bound for the RHS of (3.16), intuitively we can consider two events: the event $F_{0}$ when all of the $\alpha_{i}$’s are close to constant (defined rigorously later in equation (3.19)) and the complementary event $F_{0}^{c}$.

We claim that $\mathop{\mathbb{E}}\displaylimits\left[h(\alpha)\mathbf{1}(F_{0})\right]$ can be bounded by $2^{-d/2}$. Then we will argue that it’s difficult to get an upperbound for $\mathop{\mathbb{E}}\displaylimits\left[h(\alpha)\mathbf{1}(F_{0}^{c})\right]$ that is smaller than 1 using the RHS of equation (3.16).

It turns out in most of the calculations below, we can safely ignore the contribution of the term $3\lVert\alpha\rVert_{8}^{8}/\|\alpha\|_{6}^{6}$. For notation convenience, let $Q(\cdot):\mathbb{R}^{d}\rightarrow\mathbb{R}$ be defined as:

[TABLE]

When conditioned on the event $F_{0}$, roughly speaking, random variable $Q(\alpha)=\|\alpha\|_{4}^{4}-3\|\alpha\|^{2}$ behaves like a Gaussian distribution with variance $\Theta(n)$, since $Q(\alpha)$ is a sum of independent random variables with constant variances. Note that $\mathbf{1}(E_{1})$ and $\mathbf{1}(E_{0})$ imply that $Q(\alpha)=\lVert\alpha\rVert_{4}^{4}-3\lVert\alpha\rVert_{2}^{2}\geq(\gamma-3)\sqrt{nd}$. Therefore, $Q(\alpha)\mathbf{1}(E_{0})\mathbf{1}(E_{1})$ behaves roughly like a truncated Gaussian distribution, that is, $X\cdot\mathbf{1}(X\geq(\gamma-3)\sqrt{nd})$ where $X\sim\mathcal{N}(0,\Theta(\sqrt{n}))$. Then for sufficiently large constant $\gamma$,

[TABLE]

Moreover, recall that when the event $E_{0}$ happens, we have that $\left\lVert\alpha\right\rVert_{6}^{6}\geq 15(1-\delta)n$. Therefore putting all these bounds together into equation (3.16) with the indicator $\mathbf{1}(F_{0})$, and using the fact that $\textup{Vol}(S^{d-1})=\frac{\pi^{d/2}}{\Gamma(d/2+1)}$, we can obtain the target bound,

[TABLE]

The heavy tail problem:

Next we explain why it’s difficult to achieve a good bound from using RHS (3.16). The critical issue is that the random variable $Q(\alpha)$ has very heavy tail, and exponentially small probability cannot be ignored. To see this, we first claim that $Q(\alpha)$ has a tail distribution that roughly behaves like

[TABLE]

Taking $t=d^{2}$, then the contribution of the tail to the expectation of $Q(\alpha)^{d}$ is at least $\Pr\left[Q(\alpha)\geq t\right]\cdot t^{d}\approx d^{2d}$, which is much larger than what we obtained (equation (3.18)) in the case when all $\alpha_{i}$’s are small.

The obvious fix is to consider $Q(\alpha)^{d}(\lVert\alpha\rVert_{6}^{6})^{-d/2}$ together (instead of separately). For example, we will use Cauchy-Schwarz inequality to obtain that $Q(\alpha)^{d}(\lVert\alpha\rVert_{6}^{6})^{-d/2}\leq\lVert\alpha\rVert^{d}$. However, this alone is still not enough to get an upper bound of RHS of (3.16) that is smaller than 1. In the next paragraph, we will tighten the analysis by strengthening the estimates of $\det(M)\mathbf{1}(M\succeq 0)$.

Question II: how to tighten the estimate of $\det(M)\mathbf{1}(M\succeq 0)$?

It turns out that the AM-GM inequality is not always tight for controlling the quantity $\det(M)\mathbf{1}(M\succeq 0)$. In particular, it gives pessimistic bound when $M\succeq 0$ is not true. As a matter of fact, whenever $F_{0}^{c}$ happens, $M$ is unlikely to be positive semidefinite! (The contribution of $\det(M)\mathbf{1}(M\succeq 0)$ will not be negligible since there is still tiny chance that $M$ is PSD. As we argued before, even exponential probability event cannot be safely ignored.) We will show that formally the event $\mathbf{1}(M\succeq 0)\mathbf{1}(F_{0}^{c})$ have small enough probability that can kill the contribution of $Q(\alpha)^{d}$ when it happens (see Lemma 4.6 formally).

Summary with formal statements.

Below, we formally setup the notation and state two Propositions that summarize the intuitions above. Let $\tau=Kn/d$ where $K$ is a universal constant that will be determined later. Let $F_{k}$ be the event that

[TABLE]

We note that $1\leq\mathbf{1}(F_{0})+\mathbf{1}(F_{1})+\dots+\mathbf{1}(F_{k})$, and therefore we have

[TABLE]

Therefore towards controlling $\mathop{\mathbb{E}}\displaylimits\left[h(\alpha)\right]$, it suffices to have good upper bounds on $\mathop{\mathbb{E}}\displaylimits\left[h(\alpha)\mathbf{1}(F_{k})\right]$ for each $k$, which are summarized in the following two propositions.

Proposition 3.8**.**

Let $K\geq 2\cdot 10^{3}$ be a universal constant. Let $\tau=Kn/d$ and let $\gamma,\beta$ be sufficiently large constants (depending on $K$). Then for any $n\geq\beta d\log^{2}d$, we have that for any $k\in\{1,\dots,n\}$

[TABLE]

Proposition 3.9**.**

In the setting of Proposition 3.8, we have,

[TABLE]

We see that Theorem 3.2 can be obtained as a direct consequence of Proposition 3.9, Proposition 3.8 and Lemma 3.6.

Proof of Theorem 3.2.

Using equation (3.21) and Lemma 3.6, we have that

[TABLE]

∎

3.2 Local analysis

For a point $x$ in the local region $L_{2}^{c}$, the gradient and the Hessian are dominated by the contributions from the components that have a large correlation with $x$. Therefore it is easier to analyze the first and second order optimality condition directly.

Note that although this region is small, there are still exponentially many critical points (as an example, there is a critical point near $\frac{a_{1}+a_{2}}{\|a_{1}+a_{2}\|}$). Therefore our proof still needs to consider the second order conditions and is more complicated than Anandkumar et al. [AGJ15].

We first show that all local maxima in this region must have very high correlation with one of the components:

Lemma 3.10**.**

In the setting of Theorem 3.3, for any local maximum $x\in L_{2}$, there exists a component $a\in\{\pm a_{i}/\|a_{i}\|\}$ such that $\langle x,a\rangle\geq 0.99$.

Intuitively, if there is a unique component $a_{i}$ that is highly correlated with $x$, then the gradient $\textup{grad }f(x)=\sum_{j=1}^{n}\langle a_{j},x\rangle^{3}a_{j}$ will be even more correlated with $a_{i}$. On the other hand, if $x$ has similar correlation with two components (e.g., $x=\frac{a_{1}+a_{2}}{\|a_{1}+a_{2}\|}$), then $x$ is close to a saddle point, and the Hessian will have a positive direction that makes $x$ more correlated with one of the two components.

Once $x$ has really high correlation with one component ($|\langle x,a_{i}\rangle|\geq 0.99\|a_{i}\|$), we can prove that the gradient of $x$ is also in the same local region. Therefore by fixed point theorem, there must be a point where $\textup{grad }f(x)=x$, and it is a critical point. We also show the function is strongly concave in the whole region where $|\langle x,a_{i}\rangle|\geq 0.99\|a_{i}\|$. Therefore we know the critical point is actually a local maximum.

4 Bounding the number of local minima using Kac-Rice Formula

In this section we give the proof of Proposition 3.9 and Proposition 3.8, using the intuition described in Section 3.

4.1 Proof of Proposition 3.9

Following the plan in Section 3, we start with using AM-GM inequality for the determinant.

Lemma 4.1**.**

Under the setting of Proposition 3.9, we have,

[TABLE]

The next thing that we need to bound is $\mathop{\mathbb{E}}\displaylimits\left[|\textup{tr}(M)|^{d-1}\mid g,\alpha\right]$. Here the basic intuition is that since $\textup{tr}(M)\mid g,\alpha$ is quite concentrated around its mean and therefore we can switch the $(d-1)$-th power with the expectation without losing much. Proving such a result would require the understanding of the distribution of $M\mid g,\alpha$. We have the following generic Claim that computes this distribution in a more general setting.

Claim 4.2**.**

Let $w,y\in\mathbb{R}^{n}$ be two fixed vectors, and let $\bar{q}=y^{\odot 3}/\lVert y^{\odot 3}\rVert$. Let $B=[b_{1},\dots,b_{n}]\in\mathbb{R}^{(d-1)\times n}$ be a matrix with independent standard Gaussians as entries. Let $M_{w}=\|w\|_{4}^{4}\cdot\textup{Id}_{d-1}-3\sum_{i=1}^{n}w_{i}^{2}b_{i}b_{i}^{\top}$ and $g_{y}=\sum_{i=1}^{n}y_{i}^{3}b_{i}$. Let $Z\in\mathbb{R}^{(d-1)\times n}$ be a random matrix with rows independent drawn from $\mathcal{N}(0,\textup{Id}_{n}-\bar{q}\bar{q}^{\top})$.

Then, we have that $B\mid(g_{y}=0)$ has the same distribution as $Z$, and consequently $M_{w}\mid g_{y}=0$ has the same distribution as

[TABLE]

One can see that the setting that we are mostly interested in corresponds to $w=y=\alpha$ in Claim 4.2.

Proof.

Let $c_{1}^{\top},\dots,c_{d-1}^{\top}$ be the rows of $B$. We write $c_{i}=\beta_{i}\bar{q}+z_{i}$ where $\beta_{i}=\langle c_{i},\bar{q}\rangle$ and $z_{i}=(\textup{Id}_{n}-\bar{q}\bar{q}^{\top})c_{i}$. Let $Z$ denote the matrix with $z_{i}^{\top}$ as rows. Therefore we have $B=Z+\beta\bar{q}^{\top}$. Recall that $b_{i}$ are independent Gaussians with covariance $\mathcal{N}(0,\textup{Id}_{d-1})$. Thus we have that $c_{i}$’s are independent Gaussians with covariance $\mathcal{N}(0,\textup{Id}_{n})$, and consequently $\beta_{i}$ and $z_{i}$ are independent random variables with $\mathcal{N}(0,1)$ and $\mathcal{N}(0,\textup{Id}_{n}-\bar{q}\bar{q}^{\top})$ respectively. Moreover, note that $g=0$ is equivalent to that $\forall i\in[d-1],\langle c_{i},q\rangle=0$, which in turn is equivalent to that $\forall i\in[d-1],\beta_{i}=0$. Hence, $B\mid(g_{y}=0)$ has the same distribution as $Z$, and consequently $M_{w}\mid(g_{y}=0)$ has the same distribution as $\|w\|_{4}^{4}\textup{Id}_{d-1}-3Z\operatorname{diag}(w^{\odot 2})Z^{\top}$. ∎

Using Claim 4.2, we can compute the the expectation of the trace and its $p$-th moments. Again we state the following proposition in a more general setting for future re-usability.

Proposition 4.3**.**

In the setting of Claim 4.2, we have

[TABLE]

Moreover, for any $\varepsilon\in(0,1)$,

[TABLE]

Proof of Proposition 4.3.

We use the same notations as in the proof of Claim 4.2. By Claim 4.2, we have $B=[b_{1},\dots,b_{n}]\mid(g_{y}=0)$ has the same distribution as $Z$, and consequently $M_{w}\mid(g_{y}=0)$ has the same distribution as $\|w\|_{4}^{4}\textup{Id}_{d-1}-3Z\operatorname{diag}(w^{\odot 2})Z^{\top}$. Furthermore, this implies that $\textup{tr}(M_{w}\mid(g_{y}=0))$ has the same distribution as the random variable (denoted by $\Gamma$ below)

[TABLE]

Since $z_{k}\odot w$ is Gaussian random variable, we have that $\|z_{k}\odot w\|^{2}$ is a sub-exponential random variable with parameter $(\nu,\rho)$ satisfying $\nu\lesssim\|w\|_{4}^{2}$ and $\rho\lesssim\|w\|_{\infty}^{2}$ (see Definition B.1 for the definition of sub-exponential random variable and see Lemma B.15 for the exact calculation of the parameters for this case) . It is well known that when one takes sum of independent sub-exponential random variable, the $\rho$ parameter will be preserved and $\nu^{2}$ (which should be thought of as variance) will added up (see Lemma B.4 for details). Therefore, we have that $\Gamma$ is a sub-exponential random variable with parameter $(\nu^{*},\rho^{*})$ satisfying $\nu^{*}\lesssim\sqrt{d-1}\|w\|_{4}^{2}$ and $\rho^{*}\lesssim\|w\|_{\infty}^{2}$. Moreover, by Lemma B.15 again and the linearity of expectation, we can compute the expectation of $\Gamma$,

[TABLE]

Next the basic intuition is if $\nu^{*}$ and $\rho^{*}$ are small enough compared to the mean of $\Gamma$, then $\Gamma$ is extremely concentrated around its mean, and thus $\mathop{\mathbb{E}}\displaylimits\left[|\Gamma|^{p}\right]$ is close to $\mathop{\mathbb{E}}\displaylimits\left[\Gamma\right]^{p}$. Otherwise, the mean is more or less negligible compared to $\max\{p\lVert w\rVert_{\infty},2\sqrt{pd}\lVert w\rVert_{4}^{2}\}$, then the moment of $\Gamma$ behaves like the moments of an sub-exponential random variable with mean 0. Indeed, using basic integration, we can show (see Lemma B.10) that a sub-exponential random variable with mean $\mu$ and parameter $(\nu,b)$ satisfies that for any $\varepsilon\in(0,1)$,

[TABLE]

Applying the equation above to $\Gamma$ completes the proof. ∎

Using Proposition 4.3 with $w=y=\alpha$. we obtain the following bounds for the moments of the trace of $M$.

Corollary 4.4**.**

For any fixed $\alpha$, we have

[TABLE]

Proof.

We fix the choice of $\alpha$ and assume it satisfies event $F_{0}$. We will apply Proposition 4.3 with $w=\alpha$ and $y=\alpha$. We can verify that when $E_{0}$ and $E_{2}$ both happen, we have

[TABLE]

Therefore, taking $\varepsilon=O(\delta)$ in Proposition 4.3, we have that

[TABLE]

where $\mu=\left|\mathop{\mathbb{E}}\displaylimits\left[\textup{tr}(M)\right]\right|=(d-1)(\|\alpha\|^{4}-3\|\alpha\|^{2}+\frac{3\|\alpha\|_{8}^{8}}{\|\alpha\|_{6}^{6}})$. When $E_{0}$ happens, we have $\mu\leq(d-1)(\left\lVert\alpha\right\rVert_{4}^{4}-3\left\lVert\alpha\right\rVert^{2}+3\left\lVert\alpha\right\rVert_{\infty}^{2})\leq(d-1)(\left\lVert\alpha\right\rVert_{4}^{4}-3\left\lVert\alpha\right\rVert^{2}+O(\delta d))$. It follows that

[TABLE]

∎

Putting Corollary 4.4 and Lemma 4.1 together, we can prove Proposition 3.9.

Proof of Proposition 3.9.

Moreover, when $E_{0}$ happens, we have,

[TABLE]

It follows that

[TABLE]

For simplicity, Let $Z_{i}=(\alpha_{i}^{4}-3\alpha_{i}^{2})\mathbf{1}(|\alpha_{i}|\leq\tau^{1/4})$, and $Z=\sum_{i}Z_{i}$. We observe that $E_{1}$ and $E_{0}$ implies that $Z\geq(\gamma-3)\sqrt{nd}$. Let $\gamma^{\prime}=\gamma-3$. Thus, $Z^{d}\mathbf{1}(Z\geq{\gamma^{\prime}}\sqrt{nd})\leq(\sum_{i\in[n]}(\alpha_{i}^{4}-3\alpha_{i}^{2})\mathbf{1}(|\alpha_{i}|\leq\tau^{1/4}))^{d}\mathbf{1}(E_{1})$. Then,

[TABLE]

We have that $Z_{i}$ is a sub-exponential variable with parameter $(\sqrt{41},(4+o_{\tau}(1))\tau^{1/2})$ (by Lemma B.5). Then by Lemma B.4, we have that $\sum_{i}Z_{i}$ is sub-exponential with parameters $(\sqrt{41n},(4+o_{\tau}(1))\tau^{1/2})$. Moreover, we have $Z$ has mean $o_{d}(n)$ since $n\geq d\log d$.

Then, we use Lemma B.13 to control the moments the sub-exponential random variable $Z\cdot\mathbf{1}(|Z|\geq{\gamma^{\prime}}\sqrt{nd})$. Taking $s={\gamma^{\prime}}\sqrt{nd}$ and $\nu=\sqrt{41n}$ and $b=(4+o_{\tau}(1))\tau^{1/2}$ in Lemma B.13, we obtain that when when ${\gamma^{\prime}}\geq\max\{\sqrt{41},(1+\eta)(8+o_{\tau}(1))\sqrt{\tau d/n}\}=\max\{\sqrt{41},(1+\eta)(8+o_{\tau}(1))\sqrt{K}\}$

[TABLE]

Take $\varepsilon=1/\log d$ and $\eta=1/\log d$, we obtain that

[TABLE]

Hence, combing equation (4.7),  (4.8) and (4.9), we have,

[TABLE]

∎

4.2 Proof of Proposition 3.8

In the proof of Proposition 3.8, we will have another division of the the coordinates of $\alpha$ into two subsets $S$ and $L$ according to the magnitude of $\alpha_{i}$’s. Throughout this subsection, let $C$ be a sufficiently large constant that depends on $\delta$. Let $S=\{i:|\alpha_{i}|\leq\sqrt{C\log d}\}$ and $L=[n]\backslash S$. Thus by definition, since $n\geq d\log^{2}d$, we have that event $F_{k}$ implies that $k\in L$. But we note the threshold $\sqrt{C\log d}$ here is smaller than the threshold $\tau=Kn/d$ for defining event $F_{k}$’s. The key property that we use here is that the set $L$ have cardinality at most $d$ as long as $E_{0}$ happens. This will allows us to bound the contribution of the coordinates in $L$ in a more informative way.

Claim 4.5**.**

Let $S,L$ are defined as above with $C\geq 2/\delta$. Suppose $E_{0}$ happens, then we have $|L|\leq\delta n/\log d$.

Proof.

Assume for the sake of contradiction that $|L|>\delta n/\log d$, then we can pick a subset $L^{\prime}\subset L$ of size $\delta n/\log d$ and obtain that $\lVert\alpha_{L^{\prime}}\rVert^{2}\geq|L^{\prime}|C\log d=\delta dC\geq 2d$ which violates (3.9). ∎

For notational convenience, let $P=(2-O(\delta))\tau^{1/2}d=(2-O(\delta))\sqrt{Knd}$. Recall $\alpha_{S}$ is the restriction of vector $\alpha$ to the coordinate set $S$, and $Q(\cdot):\mathbb{R}^{d}\rightarrow\mathbb{R}$ is defined as:

[TABLE]

Define $T_{1}(\alpha),T_{2}(\alpha)$ as

[TABLE]

Therefore, since $Q(\alpha_{S})=\lVert\alpha_{S}\rVert_{4}^{4}-3\lVert\alpha_{S}\rVert^{2}$ is roughly a random variable with standard deviation $\Theta(\sqrt{n})$, for large enough constant $K$, we should think of $T_{1}(\alpha)$ and $T_{2}(\alpha)$ as exponentially small random variable. The following Lemma is the key of this section, which says that when $F_{0}$ happens, the chance of $M$ being PSD is very small.

Lemma 4.6**.**

In the setting of Proposition 3.8, let $T_{1}(\alpha),T_{2}(\alpha)$ be defined as in (4.10). Then,

[TABLE]

Proof.

We assume throughout this proof that $\alpha$ is a fixed vector and that $\alpha$ satisfies $F_{k}$ and $E_{0}$. For simplicity, define $\hat{\tau}=\sqrt{C\log d}$.

Let $q=(\alpha_{1}^{3},\dots,\alpha_{n}^{3})^{\top}\in\mathbb{R}^{n}$, and $\bar{q}=q/\|q\|$. Let $z_{1},\dots z_{d-1}$ be i.i.d Gaussian random variables in $\mathbb{R}^{d}$ with covriance $\mathcal{N}(0,\textup{Id}_{n}-\bar{q}\bar{q}^{\top})$. Let $Z$ denote the matrix with $z_{i}^{\top}$ as rows and let $v_{j}~{}(j\in[n])$ denotes the $j$-th column of $Z$. Then, by Claim 4.2, random variable $(M,\mathbf{1}(E_{2}^{\prime}))\mid g=0,\alpha)$ has the same joint distribution as $(M^{\prime},\mathbf{1}(E_{2}^{\prime\prime}))$ defined below ,

[TABLE]

Then, it suffices to bound from above the quantity $\Pr\left[M^{\prime}\succeq 0\wedge E_{2}^{\prime\prime}\right]\,.$

A technical complication here is that $v_{1},\dots,v_{n}$ are not independent random variables. We get around this issue by the following “coupling” technique which introduces some additional randomness. Let $\beta\in\mathcal{N}(0,\textup{Id}_{d-1})$ be a random variable that is independent with all the $v_{i}$’s and define $\tilde{B}=\beta\bar{q}^{\top}+Z$. We can verify that $\tilde{B}$ contains independent entries with distribution $\mathcal{N}(0,1)$. Let $\tilde{b}_{1},\dots,\tilde{b}_{n}$ be the columns of $\tilde{B}$. Let $\bar{v}_{k}=v_{k}/\lVert v_{k}\rVert$. We have that $M^{\prime}\succeq 0$ implies that

[TABLE]

We bound from above $\bar{v}_{k}^{\top}M^{\prime}\bar{v}_{k}$ by decomposing it according to the subset $S,L$.

[TABLE]

Here $A_{S},A_{L}$ are defined above in equation (4.12). By Claim 4.5, we have $|L|\leq\delta n/\log d$. By (3.9) again, we have that $E_{0}$ implies that $\lVert\alpha_{L}\rVert^{2}\leq(1+\delta)d$. Then, we have $\lVert\alpha_{L}\rVert_{4}^{4}\leq\max_{i}\alpha_{i}^{2}\cdot\lVert\alpha_{L}\rVert^{2}\leq(1+\delta)d\max_{i}\alpha_{i}^{2}=(1+\delta)d\alpha_{k}^{2}$. Moreover, when $E_{2}^{\prime\prime}$ happens, we have that $\alpha_{k}^{2}\langle v_{k},\bar{v}_{k}\rangle^{2}=\alpha_{k}^{2}\lVert\bar{v}_{k}\rVert^{2}\geq\alpha_{k}^{2}(1-\delta)d$. Thus,

[TABLE]

Then combining equation (4.11),  (4.12) and (4.13), we have that when $M^{\prime}\succeq 0$ and $E_{2}^{\prime\prime}$ both happen, it holds that $A_{S}\geq(2-O(d))\tau^{1/2}d$.

Next we consider in more detail the random variable $A_{S}$. Define random variable $Y=\bar{v}_{k}^{\top}\tilde{B}_{-k}\operatorname{diag}((\alpha_{S})_{-k})$, where $\tilde{B}_{-k}$ is the $(d-1)\times(n-1)$ sub-matrix of $\tilde{B}$ that excludes the $k$-th column of $\tilde{B}$, and $(\alpha_{S})_{-k}$ is the $n-1$-dimensional vector that is obtained from removing the $k$-th entry of $\alpha_{S}$. Note that since $k\not\in S$, we have that $\lVert Y\rVert^{2}=\sum_{i\in S}\alpha_{i}^{2}\langle v_{i},\bar{v}_{i}\rangle^{2}$, and $A_{S}=\lVert\alpha_{S}\rVert_{4}^{4}-3\lVert Y\rVert^{2}$. Thus our goal next is to understand the distribution of $Y$.

We observe that $Y\mid v_{k}$ is a Gaussian random variable since $\tilde{B}_{k}\mid v_{k}$ is a Gaussian random variable. Let $\ell_{i}\in\mathbb{R}^{n-1}$ denote the $i$-th row of the matrix $\tilde{B}$. Then $\ell_{1},\dots,\ell_{d-1}$ are i.i.d Gaussian random variable with distribution $\mathcal{N}(0,\textup{Id}_{n})$. Now we consider $\ell_{i}\mid v_{k,i}$, where $v_{k,i}$ is the $i$-th coordinates of vector $v_{k}$. Note that $v_{k,i}=z_{i,k}=((\textup{Id}-\bar{q}\bar{q}^{\top})\ell_{i})_{k}=\langle e_{k}-\bar{q}_{k}\bar{q},\ell_{i}\rangle$ (here $z_{i,k}$ denotes the $k$-th coordinate of $z_{i}$ and $e_{k}$ is the $k$-th natural basis vector in $\mathbb{R}^{n}$). Thus, conditioning on $v_{k,i}$ imposes linear constraints on $\ell_{i}$. Let $s_{k}=e_{k}-\bar{q}_{k}\bar{q}\in\mathbb{R}^{n}$ and $\bar{s}_{k}=s_{k}/\lVert s_{k}\rVert$. Then $\ell_{i}\mid v_{k,i}$ has covariance $\textup{Id}_{n}-\bar{s}_{k}\bar{s}_{k}^{\top}$. Computing the mean of $\ell_{i}\mid v_{k,i}$ will be tedious but fortunately our bound will not be sensitive to it. Now we consider random variable $\ell_{i}^{\prime}=(\ell_{i})_{-k}\operatorname{diag}((\alpha_{S})_{-k})\mid v_{k,i}$ in $\mathbb{R}^{n-1}$ where $(\ell_{i})_{-k}$ denotes the restriction of $\ell_{i}$ to the subset $[n]\backslash\{k\}$. Note that $\ell_{i}^{\prime}$ is precisely a row of $\tilde{B}_{-k}\operatorname{diag}((\alpha_{S})_{-k})$. We have $\ell_{i}^{\prime}$ is again a Gaussian random variable and its covariance matrix is $\Sigma=\operatorname{diag}((\alpha_{S})_{-k})\left(\textup{Id}_{n}-\bar{s}_{k}\bar{s}_{k}^{\top}\right)_{-k,-k}\operatorname{diag}((\alpha_{S})_{-k})$, where $\left(\textup{Id}_{n}-\bar{s}_{k}\bar{s}_{k}^{\top}\right)_{-k,-k}$ is the restriction of $\textup{Id}_{n}-\bar{s}_{k}\bar{s}_{k}^{\top}$ to the subsets $([n]\backslash\{k\})\times([n]\backslash\{k\})$. Since $\ell_{i}$’s are independent with each other, we have $Y^{\top}\mid\bar{v}_{k}=[\ell_{1}^{\prime},\dots,\ell_{d-1}^{\prime}]\cdot\bar{v}_{k}$ is Gaussian random variable with covariance $\|\bar{v}_{k}\|^{2}\Sigma=\Sigma$.

Next we apply Lemma B.7 on random variable $Y^{\top}$ and obtain that $Y^{\top}$ has concentrated norm in the sense that for any choice of $v_{k}$,

[TABLE]

Recall that $\Sigma=\operatorname{diag}((\alpha_{S})_{-k})\left(\textup{Id}_{n}-\bar{s}_{k}\bar{s}_{k}^{\top}\right)_{-k,-k}\operatorname{diag}((\alpha_{S})_{-k})$. Therefore, we have $\textup{tr}(\Sigma)=\sum_{i\in S}\alpha_{i}^{2}-\lVert\bar{s}_{k}\odot(\alpha_{S})_{-k}\rVert^{2}\geq\sum_{i\in S}\alpha_{i}^{2}-\lVert\alpha_{S}\rVert_{\infty}^{2}\geq\sum_{i\in S}\alpha_{i}^{2}-\hat{\tau}^{1/2}$. Moreover, we have that $\lVert\Sigma\rVert\leq\lVert\alpha_{S}\rVert^{2}\leq\hat{\tau}^{1/2}$ and $\textup{tr}(\Sigma^{2})\leq\lVert\alpha_{S}\rVert_{4}^{4}$. Plugging these into equation (4.14), and using the fact that $A_{S}=\lVert\alpha_{S}\rVert_{4}^{4}-3\lVert Y\rVert^{2}$, we have that for any $t\geq 0$,

[TABLE]

For simplicity, let $P=(2-O(\delta))\tau^{1/2}d-6\hat{\tau}^{1/2}=(2-O(\delta)-o(1))\tau^{1/2}d$ and $Q(\alpha)=\|\alpha_{S}\|_{4}^{4}-3\|\alpha_{S}\|^{2}$. Then we have that suppose $P>Q(\alpha)$, by taking $t=\frac{(P-Q(\alpha))^{2}}{36\lVert\alpha_{S}\rVert^{2}}$ in equation (4.15),

[TABLE]

Hence, we have

[TABLE]

∎

For notational convenience, again we define $T_{3}(\alpha)$ and $T_{4}(\alpha)$ as,

[TABLE]

Intuitively, $T_{3}$ and $T_{4}$ are the contribution of the small coordinates and large coordinates to the quantity $\mathop{\mathbb{E}}\displaylimits[\textup{tr}(M)]^{d}p_{g\mid\alpha}(0)\propto Q(\alpha)(\lVert\alpha\rVert_{6}^{6})^{-d/2}$, which is the central object that we are controlling.

Using Lemma 4.6, we show that bounding $\mathop{\mathbb{E}}\displaylimits\left[h(\alpha)\mathbf{1}(F_{k})\right]$ reduces to bounding the second moments of $T_{1}(\alpha),\dots,T_{4}(\alpha)$.

Lemma 4.7**.**

Let $T_{1}(\alpha),T_{2}(\alpha),T_{3}(\alpha),T_{4}(\alpha)$ be defined as above. In the setting of Proposition 3.8, we have,

[TABLE]

Proof of Lemma 4.7.

By AM-GM inequality (Claim 3.7) we obtain that

[TABLE]

It follows that,

[TABLE]

Here in the last step we used Cauchy-Schwarz inequality.

Let $T_{5}(\alpha)$ be defined as in equation (4.22), by Corollary 4.4

[TABLE]

Then, combining equation (4.23) and (4.5), we obtain that

[TABLE]

Here the last inequality uses Cauchy-Schwarz inequality and Holder inequality (technically, Lemma B.19 with $\eta=1/2$). Using Lemma 4.6, we obtain that

[TABLE]

Now we have that our final target can be bounded by

[TABLE]

∎

The next few Lemmas are devoted to controlling the second moments of $T_{1}(\alpha),\dots,T_{4}(\alpha)$. We start off with $T_{3}(\alpha)$.

Lemma 4.8**.**

In the setting of Lemma 4.7, we have,

[TABLE]

Proof.

Let $Z=\left\lVert\alpha_{S}\right\rVert_{4}^{4}-\left\lVert\alpha_{S}\right\rVert^{2}=\sum_{i=1}^{n}(\alpha_{i}^{4}-3\alpha_{i}^{2})\mathbf{1}(|\alpha_{i}|\leq\sqrt{C\log d})$. Then $Z$ is a sub-exponential random variable with parameters $(\sqrt{41n},4C\log d)$ (by Lemma B.5). Moreover, for sufficiently large $C$ (depending on the choice of $\delta$) we have $|\mathop{\mathbb{E}}\displaylimits\left[Z\right]|\leq\delta$. Therefore, by Lemma B.10, choosing $\varepsilon=O(\delta)$, we have

[TABLE]

Moreover, when $E_{0}$ happens, we have $\left\lVert\alpha_{S}\right\rVert_{6}^{6}\geq 15(1-\delta)n$. Hence,

[TABLE]

∎

Next we control the second moments of $T_{4}(\alpha)$.

Lemma 4.9**.**

In the setting of Lemma 4.7, we have

[TABLE]

As a direct consequence, we have

[TABLE]

Proof.

We have

[TABLE]

∎

The next two Lemmas control the second moments of $T_{1}(\alpha)$ and $T_{2}(\alpha)$.

Lemma 4.10**.**

In the setting of Lemma 4.7, we have

[TABLE]

Proof.

Since $Q(\alpha_{S})$ is a sub-exponential random variable with parameters $(\sqrt{41n},4C\log d)$ (by Lemma B.5), we have that (by Lemma B.3)

[TABLE]

∎

Lemma 4.11**.**

In the setting of Lemma 4.7, we have

[TABLE]

Proof.

Similarly to the proof of Lemma 4.10, we have that for $K\geq 400$, it holds that $\Pr\left[Q(\alpha)\geq P/2\right]\leq(0.1)^{d}$. Then it follows that

[TABLE]

∎

Finally using the four Lemmas above and Lemma 4.7, we can prove Proposition 3.8.

Proof of Proposition 3.8.

Using Lemma 4.7, Lemma 4.8, 4.9, 4.10,4.11, we have that

[TABLE]

∎

5 Local Maxima Near True Components

In this section we show that in the neighborhoods of the $2n$ true components, there are exactly $2n$ local maxima. Recall the set $L_{2}$ was defined to be the set of points $x$ that do not have large correlation with any of the components:

[TABLE]

We will show that the objective function (1.1) has exactly $2n$ local maxima in the set $L_{2}^{c}$, and they are close to the normalized version of $\pm a_{i}$’s.

Let $\bar{a}_{i}=\frac{a_{i}}{\|a_{i}\|}$ be the normalized version of $a_{i}$ on the unit sphere. We prove the following slightly stronger version of Theorem 3.3. (Note that Theorem 3.3 is a straightforward corollary of the Theorem below, since w.h.p, for every $i$, $\|a_{i}\|=\sqrt{d}\pm\widetilde{O}(1)$. )

Theorem 5.1**.**

Suppose $(d\log d)/\delta^{2}\leq n\leq d^{2}/\log^{O(1)}d$. Then, with high probability over the choice $a_{1},\dots,a_{n}$,

[TABLE]

Moreover, each of the point in $\mathcal{M}_{f}\cap L\cap L_{2}^{c}$ is $\tilde{O}(\sqrt{n/d^{3}})$-close to one of $\pm\bar{a}_{1},\dots\pm\bar{a}_{n}$.

Towards proving the Theorem, we first show that all the local minima in the set $L_{2}^{c}$ must have at least $0.99$ correlation with one of $\pm\bar{a}_{i}$ (see Lemma 5.2). Next we show in each of the regions $\langle x,\bar{a}_{i}\rangle\geq 0.99$, the objective function is strongly convex with a unique local maximum (see Lemma 5.5).

Lemma 5.2**.**

In the setting of Theorem 3.3, for any local maximum $x\in L_{2}$, there exists a component $a\in\{\pm\bar{a}_{i}\}$ such that $\langle x,a\rangle\geq 0.99$.

We can partition points in $L_{2}$ according to their correlations with the components. We say its largest correlation is the maximum of $|\langle\bar{a}_{i},x\rangle|$, and second largest correlation is the second largest among $|\langle\bar{a}_{i},x\rangle|$. For any point $x$ in $L_{2}$, it is in one of the three types 1) the largest correlation is at least $0.99$; 2) the largest correlation is at least $\sqrt{2}$ larger than the second largest correlation; 3) the largest and second largest correlations are within a factor of $\sqrt{2}$. Intuitively, points in case 1 are what we want for the Lemma, points in case 2 will have a nonzero gradient, points in case 3 will not have a negative semidefinite Hessian (hence points in cases 2 and 3 cannot be local maxima). We formalize this intuition in the following proof:

Proof of Lemma 5.2.

Let $c_{i}=\langle x,\bar{a}_{i}\rangle$ , without loss of generality, we can rearrange the $a_{i}$’s so that

[TABLE]

Let $k$ be the index such that $|c_{k}|\geq\frac{\log d}{\sqrt{d}}$ and $|c_{k+1}|<\frac{\log d}{\sqrt{d}}$. We will call components $1,2,...,k$ the large components, and the rest the small components. We assume the following events happen simultaneously (note that, with high probability, all of them are true):

$\{a_{i}\}$’s satisfy the $(d/\Delta\log n,0.01)$-RIP property for some universal constant $\Delta$. As a direct consequence, $k\ll d/\log d$ and $\sum_{i=1}^{k}c_{i}^{2}\leq(1.01)^{2}$.). (See Definition B.20) 2. 2.

For all $i$, we have $\|a_{i}\|=\sqrt{d}(1\pm o(1))$. 3. 3.

For all $i\neq j$, $|\langle a_{i},a_{j}\rangle|\leq\sqrt{d}\log d$. As a consequence, we have $|\langle\bar{a}_{i},\bar{a}_{j}\rangle|\leq(1+o(1))\frac{\log d}{\sqrt{d}}$. 4. 4.

Concentration Lemmas B.22, B.23 and B.24 hold.

We aim to show $|c_{1}|\geq 0.99$. First, if $0.99>|c_{1}|>\sqrt{2}|c_{2}|$, we will show the point has nonzero $\textup{grad }f(x)$, and hence cannot be a local maximum.

Claim 5.3**.**

If $0.99>|c_{1}|>\sqrt{2}|c_{2}|$, then $\frac{|\|P_{\bar{a}_{1}^{\perp}}\nabla f(x)\|}{|\langle\nabla f(x),\bar{a}_{1}\rangle|}<\frac{3}{4}\cdot\frac{\|\Pi_{\bar{a}_{1}^{\perp}}x\|}{|\langle x,\bar{a}_{1}\rangle|}+\frac{O(\sqrt{nd}\log^{4}d)}{|c_{1}|^{3}d^{2}}$, in particular when $|c_{1}|\leq 0.99$ we know $\textup{grad }f(x)\neq 0$.

Proof.

Let $x^{\prime}=\nabla f(x)$, we know the gradient $\textup{grad }f(x)=0$ if and only if $x^{\prime}$ is a multiple of $x$. Therefore we shall show $x^{\prime}$ is not a multiple of $x$ by showing $\frac{|\|\Pi_{\bar{a}_{1}^{\perp}}x^{\prime}\|}{|\langle x^{\prime},\bar{a}_{1}\rangle|}>\frac{\|\Pi_{\bar{a}_{1}^{\perp}}x\|}{|\langle x,\bar{a}_{1}\rangle|}$. Intuitively, after one step of gradient ascent/power iterations, the point should become more correlated with $\pm a_{1}$.

Since $x^{\prime}=\sum_{i=1}^{n}4\langle x,a_{i}\rangle^{3}a_{i}$, we can write $x^{\prime}=4\langle a_{1},x\rangle^{3}a_{1}+x^{\prime}_{L}+x^{\prime}_{S}$, where $x^{\prime}_{L}=\sum_{i=2}^{k}4\langle x,a_{i}\rangle^{3}a_{i}$ and $x^{\prime}_{S}=\sum_{i=k+1}^{n}4\langle x,a_{i}\rangle^{3}a_{i}$. The first term has norm $(c_{1}^{3})(1\pm o(1))d^{2}$ and is purely in the direction $\bar{a}_{1}$. We just need to show the other terms are small. First, we bound $\|x^{\prime}_{S}\|$ by concentration inequality Lemma B.23: for any vector $x$ we know

[TABLE]

On the other hand, we have

[TABLE]

Therefore $x^{\prime}_{L}$ does not have large correlation with $\bar{a}_{1}$. We can also bound the norm of $\Pi_{\bar{a}_{1}^{\perp}}x^{\prime}_{L}$: by RIP condition we know $\bar{a}_{1},...,\bar{a_{k}}$ form an almost orthonormal basis, therefore

[TABLE]

We know $c_{i}=\langle c_{1}\bar{a}_{1},\bar{a}_{i}\rangle+\langle\Pi_{\bar{a}_{1}^{\perp}}x,\bar{a}_{i}\rangle$, by Claim B.18 we know that $\|a+b\|_{6}^{6}\leq 1.01\|a\|_{6}^{6}+O(\|b\|_{6}^{6})$, therefore

[TABLE]

where the last inequality follows from RIP condition as it implies $\sum_{i=2}^{k}\langle\Pi_{\bar{a}_{1}^{\perp}}x,\bar{a}_{i}\rangle^{2}\leq 1.01\|\Pi_{\bar{a}_{1}^{\perp}}x\|^{2}$.

Combining these we know

[TABLE]

Therefore

[TABLE]

∎

Next we show if $|c_{1}|\leq\sqrt{2}|c_{2}|$, then the Hessian $\textup{Hess }f(x)$ is not negative semidefinite, hence again $x$ cannot be a local maximum

Claim 5.4**.**

If $|c_{1}|\leq\sqrt{2}|c_{2}|$, we have $\sigma_{max}(\textup{Hess }f(x))>0$.

Proof.

Consider the space spanned by $\bar{a}_{1}$ and $\bar{a_{2}}$, this is a two dimensional subspace so there is at least one direction that is orthogonal to $x$. Let $u$ such a direction ($u\in\mbox{span}(\bar{a}_{1},\bar{a_{2}}),\langle u,x\rangle=0,\|u\|=1$). By Claim 2.1, we know the Hessian is equal to

[TABLE]

Clearly, the first term is PSD and the second term is negative. We will consider $u^{\top}\textup{Hess }f(x)u$, and break both terms in the Hessian into the sum of large and small components. For the large components, we know $(c_{1}^{2}\|a_{1}\|^{2}a_{1}a_{1}^{\top}+c_{2}^{2}\|a_{2}\|^{2}a_{2}a_{2}^{\top})$ is a matrix that has smallest singular value at least $c_{2}^{2}d^{2}(1-o(1))$ in subspace $\mbox{span}(\bar{a}_{1},\bar{a_{2}})$ because $a_{1}$ and $a_{2}$ are almost orthogonal. Also, $u$ is orthogonal to $x$ so $P_{x}u=u$, therefore for large components

[TABLE]

On the other hand, for the second term, the contribution from large components is smaller

[TABLE]

For the small components, we only count their contributions to the second term (because the first term is positive anyways), by concentration inequality Lemma B.22, we know with high probability for all $u$

[TABLE]

The last inequality is because $c_{2}^{2}\geq c_{1}^{2}/2\geq\delta^{2}(1-o(1))/2$ and $n\ll d^{2}\delta^{2}$. Therefore, combining all these terms we know

[TABLE]

Therefore the Hessian is not negative semidefinite, and the point $x$ could not have been a local maximum. ∎

The Lemma follows immediately from the two claims.

∎

Lemma 5.5**.**

Under the same condition as Theorem 3.3, for all vector $z$ in $\{\pm\bar{a}_{i}\}$, in the set $\{x:\langle x,z\rangle\geq 0.99\}$there is a unique local maximum that is $\tilde{O}(\sqrt{n/d^{3}})$-close to $z$.

The fact that power method converges to a local maximum in this region is proven by Anandkumar et al.[AGJ15]. We give the proof for completeness.

Proof.

Without loss of generality we assume $\langle x,\bar{a}_{1}\rangle\geq 0.99$.

By Claim 5.3, we know for any $x$ in the set $\{x:\langle x,\bar{a}_{1}\rangle\geq 0.99\}$, $\frac{\nabla f(x)}{\|\nabla f(x)\|}$ is in the same set. Therefore by Schauder fixed point theorem[Wik16b] there is at least a critical point in this set. Again by Claim 5.3 we know every critical point must satisfy $\frac{|\|\Pi_{\bar{a}_{1}^{\perp}}\nabla f(x)\|}{|\langle\nabla f(x),\bar{a}_{1}\rangle|}\leq\frac{O(\sqrt{nd}\log^{4}d)}{|c_{1}|^{3}d^{2}}$. Therefore we only need to prove there is a unique critical point and it is also a local maximum. We do this by showing the Hessian $\textup{Hess }f(x)$ is always negative semidefinite in this set.

Again let $c_{i}=\langle x,\bar{a}_{i}\rangle$ and rearrange the $a_{i}$’s so that

[TABLE]

Let $k$ be the index such that $|c_{k}|\geq\frac{\log d}{\sqrt{d}}$ and $|c_{k+1}|<\frac{\log d}{\sqrt{d}}$. With high probability, we know $\{a_{i}\}$ satisfy the $(d/4,0.01)$-RIP property. Under RIP condition we know $k\ll d/\log d$ and $\sum_{i=1}^{k}c_{i}^{2}\leq(1+\zeta)^{2}$. We also condition on event that for all $i$ $\|a_{i}\|=\sqrt{d}(1\pm o(1))$ and for all $i,j$ $|\langle a_{i},a_{j}\rangle|\leq\sqrt{d}\log d$ which happens with high probability when $n\geq d\log d$.

Now consider the Hessian

[TABLE]

The second term is obviously larger than $c_{1}^{4}(1-o(1))d^{2}=(1-o(1))d^{2}$. Therefore we only need to show the spectral norm of the first term is much smaller. We can break the first term into 3 parts: the first component, components 2 to $k$, and components $k+1$ to $n$. For the first component, we know in the set $\|P_{x}a_{1}\|\leq(1+o(1))\sqrt{0.02d}$, so

[TABLE]

For the second component,

[TABLE]

Here the last inequality used the fact that $A$ is RIP and therefore $\sum_{i=2}^{k}\bar{a}_{i}\bar{a}_{i}^{\top}$ cannot have large eigenvalue. Finally for the third component we can bound it directly by concentration inequality Lemma B.24,

[TABLE]

Taking the sum of these three equations (5.4-5.6), we know $\textup{Hess }f(x)$ is always negative semidefinite in the set. Therefore the set can only have a unique critical point which is also a local maximum. ∎

6 Conclusion

We analyze the optimization landscape of the random over-complete tensor decomposition problem using the Kac-Rice formula and random matrix theory. We show that in the superlevel set $L$ that contains all the points with function values barely larger than the random guess, there are exactly $2n$ local maxima that correspond to the true components. This implies that with an initialization slight better than the random guess, local search algorithms converge to the desired solutions. We believe our techniques can be extended to 3rd order tensors, or other non-convex problems with structured randomness.

The immediate open question is whether there is any other spurious local maximum outside this superlevel set. Answering it seems to involve solving difficult questions in random matrix theory. Another potential approach to unravel the mystery behind the success of the non-convex methods is to analyze the early stage of local search algorithms and show that they will enter the superlevel set $L$ quickly.

Acknowledgements

This paper was done in part while the authors were hosted by Simons Institute. We are indebted to Ramon van Handel for very useful comments on random matrix theory. We thank Nicolas Boumal and Daniel Stern for helpful discussions. This research was supported by NSF, Office of Naval Research, and the Simons Foundation.

Appendix A Brief introduction to manifold optimization

Let $\mathcal{M}$ be a Riemannian manifold. For every $x\in\mathcal{M}$, let $T_{x}\mathcal{M}$ be the tangent space to $\mathcal{M}$ at $x$, and $P_{x}$ denotes the projection to the tangent space $T_{x}\mathcal{M}$. Let $\textup{grad }f(x)\in T_{x}\mathcal{M}$ be the gradient of $f$ at $x$ on $\mathcal{M}$ and $\textup{Hess }f(x)$ be the Riemannian Hessian. Note that $\textup{Hess }f(x)$ is a linear mapping from $T_{x}\mathcal{M}$ onto itself.

In this paper, we work with $\mathcal{M}=S^{d-1}$, and we view it as a submanifold of $\mathbb{R}^{d}$. We also view $f$ as the restriction of a smooth function $\bar{f}$ to the manifold $\mathcal{M}$. In this case, we have that $T_{x}\mathcal{M}=\{z\in\mathbb{R}^{d}:z^{\top}x=0\}$, and $P_{x}=\textup{Id}-xx^{\top}$. The gradient can be computed by,

[TABLE]

where $\nabla$ is the usual gradient in the ambient space $\mathbb{R}^{d}$. Moreover, the Riemannian Hessian is defined as,

[TABLE]

Here we refer the readers to the book [AMS07] for the derivation of these formulae and the exact definition of gradient and Hessian on the manifolds.444For example, the gradient is defined in [AMS07, Section3.6, Equation (3.31)], and the Hessian is defined in[AMS07, Section 5.5, Definition 5.5.1].  [AMS07, Example 5.4.1] gives the Riemannian connection of the sphere $S^{d-1}$ which can be used to compute the Hessian.

Appendix B Toolbox

In this section we collect most of the probabilistic statements together. Most of them follow from basic (but sometimes tedious) calculation and calculus. We provide the full proofs for completeness.

B.1 Sub-exponential random variables

In this subsection we give the formal definition of sub-exponential random variables, which is used heavily in our analysis. We also summarizes it’s properties.

Definition B.1** (c.f.[Wai15, Definition 2.2]).**

A random variable $X$ with mean $\mu=\mathop{\mathbb{E}}\displaylimits\left[X\right]$ is sub-exponential if there are non-negative parameters $(\nu,b)$ such that

[TABLE]

The following lemma gives a sufficient condition for $X$ being a sub-exponential random variable.

Lemma B.2**.**

Suppose random variable $X$ satisfies that for any $t\geq 0$,

[TABLE]

Then $X$ is a sub-exponential random variable with parameter $(\nu,b)$ satisfying $\nu\lesssim p$ and $b\lesssim q$.

The following Lemma gives the reverse direction of Lemma B.2 which controls the tail of a sub-exponential random variable.

Lemma B.3** ([Wai15, Proposition 2.2]).**

Suppose $X$ is a sub-exponential random variable with mean 0 and parameters $(\nu,b)$. Then,

[TABLE]

A summation of sub-exponential random variables remain sub-exponential (with different parameters).

Lemma B.4** (c.f. [Wai15]).**

Suppose independent random variable $X_{1},\dots,X_{n}$ are sub-exponential variables with parameter $(\nu_{1},b_{1}),\dots,(\nu_{n},b_{n})$ respectively, then $X_{1}+\dots+X_{n}$ is a sub-exponential random variable with parameter $(\mu^{*},b^{*})$ where $\nu^{*}=\sqrt{\sum_{k\in[n]}\nu_{k}^{2}}$ and $b^{*}=\max_{k}b_{k}$.

The following Lemma uses the Lemma above to prove certain sum of powers of Gaussian random variables are sub-exponential random variables.

Lemma B.5**.**

Let $x\sim\mathcal{N}(0,1)$ and $z=(x^{4}-3x^{2})\mathbf{1}(|x|\leq\tau^{1/4})$ where $\tau>C$ for a sufficiently large constant $C$. Then we have that $Z$ is a sub-exponential random variable with parameter $(\nu,b)$ satisfying $\nu=\sqrt{41}$ and $b=(4+o_{\tau}(1))\sqrt{\tau}$.

Proof.

We verify that $Z$ satisfies the Bernstein condition

[TABLE]

For $k=2$, we have that $\mathop{\mathbb{E}}\displaylimits\left[z^{2}\right]\leq\mathop{\mathbb{E}}\displaylimits\left[x^{8}-6x^{6}+9x^{4}\right]=41=\frac{1}{2}\cdot 2!\cdot\nu^{2}$. For any $k$ with $(1-o_{\tau}(1))\frac{e\sqrt{\tau}}{4}\geq k>3$, we have

[TABLE]

On the other hand, when $k\geq(1-o_{\tau}(1))\frac{e\sqrt{\tau}}{4}$, we have

[TABLE]

∎

B.2 Norm of Gaussian random variables

The following Lemma shows that the norm of a Gaussian random variable has an sub-exponential upper tail and sub-Gaussian lower tail.

Lemma B.6** ([HKZ12b, Proposition 1.1]).**

Let $A\in\mathbb{R}^{n\times n}$ be a matrix and $\Sigma=A^{\top}A$. Let $x\sim\mathcal{N}(0,\textup{Id}_{n})$. Then for all $t\geq 0$,

[TABLE]

Moreover, $\|Ax\|^{2}$ is a sub-exponential random variable with parameter $(\nu,b)$ with $\nu\lesssim\sqrt{\textup{tr}(\Sigma^{2})}$ and $b\lesssim\|\Sigma\|$.

We note that this is a slight strengthen of [HKZ12b, Proposition 1.1] with the lower tail. The strengthen is simple. The key of  [HKZ12b, Proposition 1.1] is to work in the eigenspace of $\Sigma$ and apply [LM00, Lemma 1]. The lower tail can be bounded exactly the same by using the lower tail guarantee of [LM00, Lemma 1]. The second statement follows simply from Lemma B.2.

The following Lemma is a simple helper Lemma that says that the norm of a product of a Gaussian matrix with a vector has a sub-Gaussian lower tail.

Lemma B.7**.**

Let $v\in\mathbb{R}^{m}$ be a fixed vector and $z_{1},\dots,z_{m}\in\mathbb{R}^{n}$ be independent Gaussian random with mean $\mu\in\mathbb{R}^{n}$ and covariance $\Sigma$. Then $X=[z_{1},\dots,z_{m}]v$ is a Gaussian random variable with mean $(v^{\top}\mathbf{1})\mu$ and covariance $\|v\|^{2}\Sigma$.

Moreover, we have that for any $t\geq 0$,

[TABLE]

Proof.

The mean and variance of $X$ can be computed straightforwardly. Towards establishing (B.3), let $\bar{\mu}=\mu/\lVert\mu\rVert$, and we define $X^{\prime}=(\textup{Id}-\bar{\mu}\bar{\mu}^{\top})X$. Thus we have $\lVert X\rVert\geq\lVert X^{\prime}\rVert$ and that $X^{\prime}$ is Gaussian random variable with mean 0 and covariance $\Sigma^{\prime}=(\textup{Id}-\bar{\mu}\bar{\mu}^{\top})\Sigma(\textup{Id}-\bar{\mu}\bar{\mu}^{\top})$. Then by Lemma B.6, we have that

[TABLE]

Finally we observe that $\textup{tr}{\Sigma}\geq\textup{tr}(\Sigma^{\prime})\geq\textup{tr}(\Sigma)-\lVert\Sigma\rVert$ and this together with equation above completes the proof. ∎

B.3 Moments of sub-exponential random variables

In this subsection, we give several Lemmas regarding the estimation of the moments of (truncated) sub-exponential random variables.

We start with a reduction Lemma that states that one can control the moments of sub-exponential random variable with the moments of the Gaussian random variable and exponential random variables.

Lemma B.8**.**

Suppose $X$ is a sub-exponential random variable with mean 0 and parameters $(\nu,b)$. Let $X_{1}\sim\mathcal{N}(0,\nu^{2})$ and $X_{2}$ be a exponential random variable with mean $2b$. Let $p\geq 1$ be an integer and $s\geq 0$. Then,

[TABLE]

Proof.

Let $p(x)$ be the density of $X$ and let $G(x)=\Pr[X\geq x]$. By Lemma B.3 we have $G(x)\leq e^{-x^{2}/(2\nu^{2})}+e^{-x/(2b)}$. Then we have

[TABLE]

∎

Next we bound the $p$-th moments of a sub-exponential random variable with zero mean.

Lemma B.9**.**

Let $p$ be an even integer and let $X$ be a sub-exponential random variable with mean 0 and parameters $(\nu,b)$. Then we have that

[TABLE]

Proof.

Let $X_{1}\sim\mathcal{N}(0,\nu^{2})$ and $X_{2}$ be a exponential random variable with mean $2b$. By Lemma B.8, we have that

[TABLE]

For the negative part we can obtain the similar bound, $\mathop{\mathbb{E}}\displaylimits\left[(-X)^{p}\mathbf{1}(X\leq 0)\right]\lesssim\nu^{p}p!!+(2b)^{p}p!$. These together complete the proof. ∎

The Lemma above implies the bound for the moments of sub-exponential variable with arbitrary mean via Holder inequality.

Lemma B.10**.**

Let $X$ be a sub-exponential random variable with mean $\mu$ and parameters $(\nu,b)$. Let $p$ be an integer. Then for any $\varepsilon\in(0,1)$ we have,

[TABLE]

Proof.

Write $X=\mu+Z$ where $Z$ has mean 0. Then we have that for any $\varepsilon\in(0,1)$, by Holder inequality,

[TABLE]

∎

The next few Lemmas are to bound the moments of a truncated sub-exponential random variable. We use Lemma B.8 to reduce the estimates to Gaussian and exponential cases. Thus, we first bound the moment of a truncated Gaussian.

Claim B.11**.**

Suppose $X\sim\mathcal{N}(0,1)$. Then, as $p\rightarrow\infty$, we have for $s\geq\sqrt{p}$, $\mathop{\mathbb{E}}\displaylimits\left[|X|^{p}\mathbf{1}(|X|\geq s)\right]\leq(1+o_{p}(1))^{p}\cdot p\cdot e^{-s^{2}/2}s^{p}$

Proof.

We assume $p$ to be an odd number for simplicity. The even case can be bounded by the odd case straightforwardly. Let $\Gamma(\cdot,\cdot)$ be the upper incomplete Gamma function, we have,

[TABLE]

∎

The next Claim estimates the moments of a truncated exponential random variable.

Claim B.12**.**

Suppose $X$ has exponential distribution with mean 1 (that is, the density is $p(x)=e^{-x}$). Then, as $p\rightarrow\infty$, for any $s\geq(1+\varepsilon)p$ and $\varepsilon\in(0,1)$, we have $\mathop{\mathbb{E}}\displaylimits\left[X^{p}\mathbf{1}(X\geq s)\right]\lesssim e^{-s}s^{p}/\varepsilon$

Proof.

We have that

[TABLE]

∎

Combing the two Lemmas and use Lemma B.8 we obtain the bounds for sub-exponential random variables.

Lemma B.13**.**

Suppose $X$ is a sub-exponential random variable with mean $\mu$ and parameters $(\nu,b)$. Then, for integer $p\geq 2$, real numbers $\eta,\varepsilon\in(0,1)$ and $s\geq\max\{\nu\sqrt{p},2bp(1+\eta)\}$, we have

[TABLE]

where $\zeta_{p}\rightarrow 0$ as $p\rightarrow\infty$.

Proof.

Write $X=\mu+Z$ where $Z$ has mean 0. Then we have that for any $\varepsilon\in(0,1)$, by Holder inequality,

[TABLE]

Therefore it remains to bound $\mathop{\mathbb{E}}\displaylimits\left[|Z|^{p}\right]$. Let $p(z)$ be the density of $Z$. Since $Z$ is sub-exponential, we have that $p(z)\leq\max\{e^{-z^{2}/(2\nu^{2})},e^{-z/(2b)}\}\leq e^{-z^{2}/(2\nu^{2})}+e^{-z/(2b)}$. Let $Z_{1}\sim\mathcal{N}(0,\nu^{2})$ and $Z_{2}$ be an exponential random variable with mean $2b$. Then by Lemma B.8, we have

[TABLE]

Combing inequality above and equation (B.5) we complete the proof. ∎

Finally, the following is a helper claim that is used in the proof of Claim B.12.

Claim B.14**.**

For any $\mu\in\mathbb{R}$ and $\beta>0$, we have that

[TABLE]

Proof.

Let $F(x)$ denote the RHS of (B.6). We differentiate $F(x)$ and obtain that

[TABLE]

Therefore, the claim follows from integration both hands of the equation above. ∎

B.4 Auxiliary Lemmas

Lemma B.15**.**

Let $\bar{\tau}$ be unit vector in $\mathbb{R}^{n}$, $w\in\mathbb{R}^{n}$, and $z\sim\mathcal{N}(0,\sigma^{2}\cdot(\textup{Id}_{n}-\bar{q}\bar{q}^{\top}))$. Then $\lVert z\odot w\rVert^{2}$ is a sub-exponential random variable with parameter $(\nu,b)$ satisfying $\nu\lesssim\sigma^{2}\|w\|_{4}^{2}$ and $b\lesssim\sigma^{2}\|w\|_{\infty}^{2}$. Moreover, it has mean $\sigma^{2}(\|w\|^{2}-\langle\bar{q}^{\odot 2},w^{\odot 2}\rangle)$.

Proof.

We have that $z\odot w$ is a Gaussian random variable with covariance matrix $\Sigma=\sigma^{2}P_{\bar{q}}\operatorname{diag}(w^{\odot 2})P_{\bar{q}}$. Then, $z\odot w$ can be written as $\Sigma^{1/2}x$ where $x\sim\mathcal{N}(0,\textup{Id}_{n})$. Then by Lemma B.6, we have that $\|z\odot w\|^{2}=\|\Sigma^{1/2}x\|^{2}$ is a sub-exponential random variable with parameter $(\nu,b)$ satisfying

[TABLE]

Here at the second and forth line we both used the fact that $\textup{tr}(PA)\leq\textup{tr}(A)$ holds for any symmetric PSD matrix and projection matrix $P$.

Moreover, we have $b\lesssim\|\Sigma\|\leq\sigma^{2}\max_{k}w_{k}^{2}$. Finally, we have the mean of $\|z\odot w\|^{2}$ is $\textup{tr}(\Sigma)=\sigma^{2}(\|w\|^{2}-\langle\bar{q}^{\odot 2},w^{\odot 2}\rangle)$, which completes the proof. ∎

Claim B.16** (folklore).**

We have that $e\left(\frac{n}{e}\right)^{n}\leq n!\leq e\left(\frac{n+1}{e}\right)^{n+1}$. It follows that

[TABLE]

Claim B.17**.**

For any $\varepsilon\in(0,1)$ and $A,B\in\mathbb{R}$, we have $(A+B)^{d}\leq\max\{(1+\varepsilon)^{d}|A|^{d},(1/\varepsilon+1)^{d}|B|^{d}\}$

Claim B.18**.**

For any vectors $a,b$, and any $\varepsilon>0$, we have $\|a+b\|_{6}^{6}\leq(1+\varepsilon)\|a\|_{6}^{6}+O(1/\varepsilon^{5})\|b\|_{6}^{6}$.

Proof.

By Cauchy-Schwartz we know we only need to prove this when $a,b$ are real numbers instead of vectors. For numbers $a,b$, $(a+b)^{6}=a^{6}+6a^{5}b+15a^{4}b^{2}+20a^{3}b^{3}+15a^{2}b^{4}+6ab^{5}+b^{6}$. All the intermediate terms $a^{i}b^{j}(i+j=6)$ can be bounded by $\varepsilon/56\cdot a^{6}+O(1/\varepsilon^{5})b^{6}$, therefore we know $(a+b)^{6}\leq(1+\varepsilon)a^{6}+O(1/\varepsilon^{5})b^{6}$. ∎

Lemma B.19**.**

For any vector $z\in\mathbb{R}^{n}$, and any partition of $[n]$ into two subsets $S$ and $L$, and any $\eta\in(0,1)$, we have that,

[TABLE]

Proof.

By Cauchy-Schwarz inequality, we have that

[TABLE]

Therefore, we obtain that

[TABLE]

∎

B.5 Restricted Isometry Property and Corollaries

In this section we will describe the Restricted Isometry Property (RIP), which was introduced in [CT05]. This is the crucial tool that we used in Section 5.

Definition B.20**.**

[RIP property] Suppose $A\in\mathbb{R}^{d\times n}$ is a matrix whose columns have unit norm. We say the matrix $A$ satisfies the $(k,\delta)$-RIP property, if for any subset $S$ of columns of size at most $k$, let $A_{S}$ be the $d\times|S|$ matrix with columns in $S$, we have

[TABLE]

In our case the random components $a_{i}$’s do not have unit norm, we abuse notation and say $A$ is RIP if the column normalized matrix satisfy the RIP property.

Intuitively RIP condition means the norm of $Ax$ is very close to $\|x\|$, if the number of nonzero entries in $x$ is at most $k$. From we know a random matrix is RIP with high probability:

Theorem B.21** ([Can08]).**

For any constant $\delta>0$, when $k=d/\Delta\log n$ for large enough constant $\Delta$, with high probability a random Gaussian matrix is $(k,\delta)$-RIP.

In our analysis, we mostly use the RIP property to show any given vector $x$ cannot have large correlation with many components $a_{i}$’s.

B.6 Concentration Inequalities

We first show if we only look at the components with small correlations with $x$, the function value cannot be much larger than $3n$.

Lemma B.22**.**

With high probability over $a_{i}$’s, for any unit vector $x$, for any threshold $\tau>1$

[TABLE]

Proof.

Let $X_{i}=\mathbf{1}(|\langle a_{i},x\rangle|\leq\tau)\langle a_{i},x\rangle^{4}-\mathbb{E}[\mathbf{1}(|\langle a_{i},x\rangle|\leq\tau)\langle a_{i},x\rangle^{4}]$. Notice that $\mathbb{E}[\mathbf{1}(|\langle a_{i},x\rangle|\leq\tau)\langle a_{i},x\rangle^{4}]\leq\mathbb{E}[\langle a_{i},x\rangle^{4}]=3$. Clearly $\mathbb{E}[X_{i}]=0$ and $X_{i}\leq\tau^{4}$. We can apply Bernstein’s inequality, and we know for any $x$,

[TABLE]

When $t\geq C((\sqrt{nd}+d\tau^{4})\log d)$ for large enough constant, the probability is smaller than $\exp(-C^{\prime}d\log d)$ for large constant $C^{\prime}$, so we can union bound over all vectors in an $\varepsilon$-net with $\varepsilon=1/d^{3}$. This is true even if we set the threshold to $2\tau$.

For any $x$ not in the $\varepsilon$-net, let $x^{\prime}$ be its closest neighbor in $\varepsilon$-net, when $a_{i}$’s have norm bounded by $O(\sqrt{d})$ (which happens with high probability), it is easy to show whenever $|\langle x,a_{i}\rangle|\leq\tau$ we always have $|\langle x^{\prime},a_{i}\rangle|\leq 2\tau$. Therefore the sum for $x$ cannot be much larger than the sum for $x^{\prime}$. ∎

Using very similar techniques we can also show the contribution to the gradient for these small components is small

Lemma B.23**.**

With high probability over $a_{i}$’s, for any unit vector $x$, for any threshold $\tau>1$

[TABLE]

Proof.

Let $v=\sum_{i=1}^{n}\mathbf{1}(|\langle a_{i},x\rangle|\leq\tau)\langle a_{i},x\rangle^{3}a_{i}$. First notice that the inner-product $\langle v,x\rangle$ is exactly the sum we bounded in Lemma B.22. Therefore we only need to bound the norm in the orthogonal direction of $x$. Since $a_{i}$’s are Gaussian we know $\langle a_{i},x\rangle$ and $P_{x}a_{i}$ are independent. So we can apply vector Bernstein’s inequality, and we know

[TABLE]

By the same $\varepsilon$-net argument as Lemma B.22 we can get the desired bound. ∎

Finally we prove similar conditions for the tensor $T$ applied to the vector $x$ twice.

Lemma B.24**.**

With high probability over $a_{i}$’s, for any unit vector $x$, for any threshold $\tau>1$

[TABLE]

Proof.

Let $H=\sum_{i=1}^{n}\mathbf{1}(|\langle a_{i},x\rangle|\leq\tau)\langle a_{i},x\rangle^{2}a_{i}a_{i}^{\top}$. First notice that the quadratic form $x^{\top}Hx$ is exactly the sum we bounded in Lemma B.22. We will bound the spectral norm in the orthogonal direction of $x$, and use the fact that $H$ is a PSD matrix, so $\|H\|\leq 2(x^{\top}Hx+\|P_{x}HP_{x}|)$ (this is basically the fact that $(a+b)^{2}\leq 2a^{2}+2b^{2}$). Since $a_{i}$’s are Gaussian we know $\langle a_{i},x\rangle$ and $P_{x}a_{i}$ are independent. So we can apply matrix Bernstein’s inequality, because $\mathbb{E}[P_{x}a_{i}a_{i}^{\top}P_{x}]=P_{x}$, we know

[TABLE]

By the same $\varepsilon$-net argument as Lemma B.22 we can get the desired bound. ∎

Appendix C Missing proofs in Section 3

Proof of Lemma 3.4.

Let $A=[a_{1},\dots,a_{n}]$. First of all by standard random matrix theory (see, e.g.,  [Section 3.3][LT13] or  [RV10, Theorem 2.6]), we have that with high probability, $\left\lVert A\right\rVert\leq\sqrt{n}+1.1\sqrt{d}$. Therefore, we have that with high probability $\sum\langle a_{i},x\rangle^{2}\leq\left\lVert A\right\rVert\leq n+3\sqrt{nd}$. By the RIP property of random matrix $A$ (see [BDDW08, Theorem 5.2] or Theorem B.21), we that with high probability, equation (3.4) holds.

Finally, using a straightforward $\varepsilon$-net argument and union bound, we have that for every unit vector $x$, $\sum_{i=1}^{n}\langle a_{i},x\rangle^{4}\geq 15n-O(\sqrt{nd}\log d)$. Therefore, when $1/\delta^{2}\cdot d\log^{2}d\leq n$, equation (3.5) also holds. ∎