Global Guarantees for Enforcing Deep Generative Priors by Empirical Risk

TL;DR

This paper provides the first theoretical guarantees that enforce deep generative priors via empirical risk minimization have no spurious stationary points, explaining their empirical success in solving inverse problems.

Contribution

It establishes that under certain conditions, the non-convex optimization landscape is free of spurious stationary points, ensuring reliable solutions for deep generative prior enforcement.

Findings

No spurious stationary points outside neighborhoods of solutions.

High probability of descent directions away from solutions.

Global geometry guarantees for non-convex inverse problems.

Abstract

We examine the theoretical properties of enforcing priors provided by generative deep neural networks via empirical risk minimization. In particular we consider two models, one in which the task is to invert a generative neural network given access to its last layer and another in which the task is to invert a generative neural network given only compressive linear observations of its last layer. We establish that in both cases, in suitable regimes of network layer sizes and a randomness assumption on the network weights, that the non-convex objective function given by empirical risk minimization does not have any spurious stationary points. That is, we establish that with high probability, at any point away from small neighborhoods around two scalar multiples of the desired solution, there is a descent direction. Hence, there are no local minima, saddle points, or other stationary points outside these neighborhoods. These results constitute the first theoretical guarantees which establish the favorable global geometry of these non-convex optimization problems, and they bridge the gap between the empirical success of enforcing deep generative priors and a rigorous understanding of non-linear inverse problems.

Equations

$$x_0\in {\mathbb{R}}^k,A\in {\mathbb{R}}^{m\times n},W_i\in {\mathbb{R}}^{n_i\times n_{i-1}}\text{ for }i=1\dots d,$$

$$G(x)=\mathrm{relu}(W_d\dots \mathrm{relu}(W_2\mathrm{relu}(W_1{x}_0))\dots ),$$

$$y_0=G(x_0),$$

$$W_1\dots W_d,A,\text{ and observations }A{y}_0,$$

$$x_0.$$

$$\displaystyle f(x):=\frac{1}{2}\Bigl{\|}AG(x)-Ay_{0}\Bigr{\|}_{2}^{2}.$$

$$D_{-v_{x,x_0}}f(x)<0,$$

$$D_{v}f(0)<0,$$

$$v_{x,x_0}$$

$$\displaystyle\Bigl{\|}\sum_{i=1}^{n}1_{w_{i}\cdot x>0}1_{w_{i}\cdot y>0}\cdot w_{i}w_{i}^{t}-Q_{x,y}\Bigr{\|}\leq\epsilon,\text{ with }Q_{x,y}=\frac{\pi-\theta_{0}}{2\pi}I_{k}+\frac{\sin\theta_{0}}{2\pi}M_{\hat{x}\leftrightarrow\hat{y}},$$

$$\displaystyle\Bigl{|}\Bigl{\langle}A\bigl{(}G(x_{1})-G(x_{2})\bigr{)},A\bigl{(}G(x_{3})-G(x_{4})\bigr{)}\Bigr{\rangle}$$

$$\le ϵ\Vert G(x_1)-G(x_2){\Vert }_2\Vert G(x_3)-G(x_4){\Vert }_2.$$

$$\displaystyle D_{-v_{x,x_{0}}}f(x)<{\color[rgb]{0,0,0}-K_{3}\frac{\sqrt{\epsilon}d^{3}}{2^{d}}\max(\|x\|_{2},\|x_{0}\|_{2})},\quad$$

$$\displaystyle D_{y}f(0)<{\color[rgb]{0,0,0}-\frac{1}{8\pi 2^{d}}\|x_{0}\|_{2}},$$

$$\displaystyle\mathbb{P}\bigl{(}|\|Ax\|_{2}^{2}-\|x\|_{2}^{2}|\geq\epsilon\|x\|_{2}^{2}\bigr{)}\leq 2e^{-mc_{0}(\epsilon)},$$

$$W_{+,x}=\mathrm{diag}(Wx>0)W.$$

$$W_{i,+,x}=\mathrm{diag}(W_i{W}_{i-1,+,x}\cdots W_{2,+,x}{W}_{1,+,x}x>0)W_i.$$

$$\displaystyle g(\theta):=\cos^{-1}\Bigl{(}\frac{(\pi-\theta)\cos\theta+\sin\theta}{\pi}\Bigr{)}.$$

$$\nabla f(x)=({\Pi }_{i=d}^1{W}_{i,+,x}{)}^t{A}^{t}A({\Pi }_{i=d}^1{W}_{i,+,x})x-({\Pi }_{i=d}^1{W}_{i,+,x}{)}^t{A}^{t}A({\Pi }_{i=d}^1{W}_{i,+,x_0})x_0.$$

$$v_{x,x_0}$$

$${\overline{v}}_{x,x_0}$$

$$|{\theta }_1-g({\theta }_0)|\le 4\sqrt{ϵ},$$

$$\Biggl{|}\cos\theta_{1}-\frac{(\pi-\theta_{0})\cos\theta_{0}+\sin\theta_{0}}{\pi}\Biggr{|}\leq 5\epsilon.$$

$${\delta }_1$$

$${\delta }_2$$

$${\delta }_3$$

$$\cos {\theta }_1$$

$$=\frac{⟨x,W_{+,x}^t{W}_{+,y}y⟩}{\sqrt{⟨x,W_{+,x}^t{W}_{+,x}x⟩⟨y,W_{+,x}^t{W}_{+,x}y⟩}}$$

$$=\frac{⟨x,Q_{x,y}y⟩+{\delta }_1}{\frac{1}{2}\sqrt{(1+2{\delta }_2)(1+2{\delta }_3)}}$$

$$|\cos {\theta }_1-2⟨x,Q_{x,y}y⟩|$$

$$\le \left|1-\frac{1}{(1-2ϵ)}\right|+2ϵ\frac{1}{(1-2ϵ)}$$

$$\le 5ϵ$$

$$v_{x,x_0}=({\Pi }_{i=d}^1{W}_{i,+,x}{)}^t{A}^{t}A({\Pi }_{i=d}^1{W}_{i,+,x})x-({\Pi }_{i=d}^1{W}_{i,+,x}{)}^t{A}^{t}A({\Pi }_{i=d}^1{W}_{i,+,x_0})x_0.$$

$$\tilde{h}_{x,y}=\frac{1}{2^{d}}\left[\Bigl{(}\prod_{i=0}^{d-1}\frac{\pi-\overline{\theta}_{i}}{\pi}\Bigr{)}y+\sum_{i=0}^{d-1}\frac{\sin\overline{\theta}_{i}}{\pi}\Bigl{(}\prod_{j=i+1}^{d-1}\frac{\pi-\overline{\theta}_{j}}{\pi}\Bigr{)}\frac{\|y\|_{2}}{\|x\|_{2}}x\right],$$

$$\Vert ({\Pi }_{i=d}^1{W}_{i,+,x}{)}^t({\Pi }_{i=d}^1{W}_{i,+,y})y-{\tilde{h}}_{x,y}{\Vert }_2$$

$$\displaystyle\bigl{\langle}(\Pi_{i=d}^{1}W_{i,+,x})x,(\Pi_{i=d}^{1}W_{i,+,y})y\bigr{\rangle}$$

Full text

1 Introduction

Exploiting the structure of images and natural signals has proven to be a fruitful endeavor across many domains of science. For instance, the wavelet transform, discovered by Daubechies and others [18], led to the observation that natural images are sparse in the wavelet basis, enabling compression algorithms such as JPEG 2000 to tame the storage and transfer of the modern deluge of image and video data. Principles of wavelet based image compression, combined with surprising advances in convex relaxation, have also opened the door to greatly improved signal acquisition strategies, which unlocked critical applications throughout the imaging sciences. In particular, breaking with the dogma of the Nyquist sampling theorem, which stems from worst-case analysis, Candes and Tao, and Donoho [12, 19, 20], provided a theory and practice of compressed sensing (CS), which exploits the sparsity of natural signals in the wavelet basis to design acquisition strategies with drastically lower sample complexity — that on par with the sparsity level of the signal at hand. In particular, using the standard basis in lieu of the wavelet basis without loss of generality, they established that to recover a vector $x\in\mathbb{R}^{n}$ with $k<n$ non-zero entries from $m=O(k\log n)$ observations $\langle x,a_{i}\rangle,i=1,2\ldots,m$, where $a_{i}$ are i.i.d Gaussian, it suffices to minimize $\|x\|_{1}$ subject to the observations with high probability. On a practical level, compressed sensing has lead to significant reduction in the sample complexity of signal acquisition of natural images, for instance speeding up MRI imaging by an order of magnitude [40]. Beyond MRI, compressed sensing has impacted many if not all imaging sciences, by providing a general tool to exploit the parsimony of natural signals to improve acquisition speed, increase SNR and reduce sample complexity. More broadly, the principled use of sparsity as a prior has led to the development of the field of matrix completion [11], breakthroughs in phase retrieval [13, 9] and blind deconvolution [2]; and is at this point routinely utilized across applied mathematics and machine learning.

Meanwhile, the advent of practical deep learning [22] has significantly improved machine understanding of image and audio data. For instance, deep learning techniques are now the state of the art across most of computer vision and have taken the field far beyond where it stood just a few years prior. The success of deep learning ostensibly stems from its ability to exploit the hierarchical nature of images and other natural signals without explicit hand-engineering. There are many techniques and add-on architectural choices associated with deep learning, but many of them are non-essential from a theoretical and, to a large extent, practical perspective, with simple convolutional deep nets with Rectified Linear Units (ReLUs) achieving close to the state of the art performance on many tasks [62]. The class of functions represented by such deep networks is readily interpretable as hierarchical compression schemes with exponentially many linear filters, each being a linear combination of filters in earlier layers. Constructing such compression schemes by hand would be quite tedious, if not impossible, and the biggest surprise and advantage of deep learning is that simple stochastic gradient descent (SGD) allows one to efficiently traverse this class of functions subject to potentially highly non-convex learning objectives. While this latter property has been empirically established in an impressive number of applications, it has so far eluded a completely satisfactory theoretical explanation.

In essence, compressive sensing, and its numerous extensions, consist of enforcing a sparsity prior to regularize the solution of an inverse problem. Thus, improvements in the state of the art of compressed sensing can come from better reconstruction algorithms, better design of signal measurements, or more sophisticated priors. Virtually all of the tens of thousands of research articles in the umbrella field of compressive sensing have focused on the first two directions, taking the linear sparsity model as the de-facto prior for regularization. Those two directions are fundamentally limited in that no approach at recovering a $k$-sparse signal with respect to a basis could succeed with fewer than $k$ measurements.

Meanwhile, there have been great strides in generative modeling of images in modern machine learning that go well beyond linear sparsity models. Such improvements in priors on natural images beyond wavelet based approaches, when properly enforced, should enable more aggressive regularization of inverse problems, leading to lower sample complexity and higher SNR than traditional compressed sensing approaches. In order to understand the potential for improving upon traditional compressive sensing, broadly speaking, as a function of advances in generative modeling, it is useful to reinterpret compressive sensing from the perspective of the field of generative modeling, a popular framework in machine learning which strives to sample from the probability distribution of natural images and other signals. Note that there is a duality between generative modeling and compression. Any compression scheme implicitly defines a generative model, by its inverse, and vice versa. In particular, wavelet based compression schemes implicitly define a generative model which attempts to sample from natural images via random sparse linear combinations of wavelet basis images. This wavelet-based generative model is clearly too loose to capture the rich hierarchical structure of natural images, making it a sufficiently expressive yet very naive prior.

Generative modeling has a rich history in machine learning, but only recent deep neural network based approaches to generative modeling have enabled the generation of realistic synthetic images in a variety of domains, for example by training generative adversarial networks (GANs) to find a Nash equilibrium of a non-convex game [23, 25]; by training variational auto-encoders (VAEs) [31, 57]; and by training autoregressive models like PixelCNN [54], which generate pixels one-at-a-time by sampling from appropriate conditional probability distributions. GANs and VAEs map a low dimensional latent code space to a higher dimensional embedding space of images or other natural signals. For instance, if we equip the latent code space with a Gaussian distribution, the goal of generative adversarial training is to produce a deep neural network generator whose push-forward distribution is the distribution of natural images or another class of natural signals. Impressively, in-between the original posting of this paper and the current version of the manuscript, deep generative modeling has advanced to the point of producing high-resolution synthetic, yet extremely photorealistic, images of celebrity faces [29]. Further, continuous motion in the latent code space of the associated deep generative models has allowed for interpolation and continuous deformation of the resulting faces, even exhibiting equivariant properties with arithmetic operations in the latent code space corresponding to semantically meaningful image variations [56].

The scope of application of deep generative modeling to regularizing inverse problems is vast. These more sophisticated priors are recently emerging in empirical applications of many fields of imaging, such as medical imaging [70, 30, 24, 55, 17, 1, 49, 68, 53, 43, 32, 42, 16, 69], microscopy [59], inpainting [45, 71], superresolution [60, 33, 28], compressed sensing [38, 37, 51, 50], image manipulation [73], and many more. See [39] for a review of deep learning for inverse problems in imaging. Importantly, approaches that regularize inverse problems using deep generative models, have empirically been shown to improve over sparsity-based approaches, advancing the state of the art in several fields. For instance, in Magnetic Resonance Imaging, deep generative networks have enabled image reconstruction that is qualitatively of higher diagnostic quality and higher SNR than traditional compressive sensing allows, and is additionally two orders of magnitude faster than sparsity-based approaches due to the utilization of GPUs in applying convolutional neural networks [46, 47]. This development is significant because of the tremendous potential clinical applications of diagnostic-quality real-time MRI visualization. Deep generative models have also empirically been used directly for compression [58]. In the case of compressed sensing, optimization of an empirical risk objective over the latent code space has been empirically shown to recover images from 10x fewer linear compressive measurements than sparsity-based approaches [8].

As the quality and reach of deep generative modeling continues to increase, signal recovery in many scenarios will benefit analogously.

As with the rest of machine learning, in the field of deep generative modeling for regularizing inverse problems, or as we refer to it the field of deep compressive sensing, empirics is far ahead of the state of theoretical justification. In this paper we initiate the rigorous study of enforcing deep generative models as priors on the solutions to inverse problems, by providing a theory of compressive sensing that goes beyond linear sparsity and into the realm of applying deep neural network based generative priors. In particular we show that under suitable randomness assumptions on the weights of a neural network and successively expansive hidden layer sizes, the empirical risk objective for recovering a latent code in $\mathbb{R}^{k}$ from $m$ linear observations of the last layer of a generative network, where $m$ is proportional to $k$ up to log factors, has no spurious local minima or saddle points, in that there is a descent direction everywhere except possibly small neighborhoods around two scalar multiples of the desired solution. Our descent direction analysis is constructive; based on deterministic conditions on the neural network weights and the measurements; and relies on novel concentration bounds of certain random matrices, uncovering some interesting geometric properties of the landscapes of empirical risk objective functions for random generative multilayer networks with ReLU activations. For a generative network that achieves a greater degree of compression, the proposed scheme would enable lower sample complexity and higher SNR. If a generative model can compress a signal to a latent code dimensionality $k$ much less then the signal’s sparsity level, then compressed sensing with the generative prior may significantly outperform compressed sensing with sparsity prior in terms of sample complexity.

1.1 Related theoretical work

Latent code space optimizations after neural network training, and the optimization over the weights of a neural network during training, may both be interpreted as inverse problems [44]. The tools developed in this paper, such as the novel nonasymptotic concentration results for high dimensional Gaussians followed by a ReLU, may be of independent interest, in particular being amenable for establishing global non-asymptotic analysis regarding convergence of SGD for training deep neural networks. Our work also relates to recent trends in optimization. Traditionally, rigorous understanding of inverse problems has been limited to the simpler setting in which the optimization objective is convex. More recently, there has been progress in understanding non-convex optimization objectives for inverse problems, in albeit analytically simpler situations than those involving multilayer neural networks. For instance, the authors of [63, 5] provide a global analysis of non-convex objectives for phase retrieval and community detection, respectively, ruling out adversarial geometries in these scenarios for the purposes of optimization. Additionally, rigorous guarantees of nonconvex recovery include other results in phase retrieval [10, 14], blind deconvolution [34, 41, 26], robust subspace recovery [48], discrete joint alignment [15], and more.

In related work, the authors of [8] also study inverting compressive linear observations under generative priors, by proving a restricted eigenvalue condition on the range of the generative neural network. However, they only provide a guarantee that is local in nature, in showing the global optimum of empirical risk is close to the desired solution. The work provides no guarantees about why the global minimum of the nonconvex problem can be reached. In addition, [4] studied inverting neural networks given access to the last layer using an analytical formula that approximates the inverse mapping of a neural network. The results of [4] are in a setting where the neural net is not generative, and their procedure is at only approximate, and, since it requires observation of the last layer, it is not readily extendable to the compressive linear observation setting. Meanwhile, the optimization problem we study can yield exact recovery, which we observe empirically via gradient descent. Most importantly, in contrast to [8, 4], we provide a global analysis of the non-convex empirical risk objective function and constructively exhibit a descent direction at every point outside a neighborhood of the desired solution and a negative scalar multiple of it. Our guarantees are non-asymptotic, and to the best of our knowledge the first of their kind.

1.2 Notation

Before we present the main result, we now introduce notation that will be used throughout this paper. Let $[n]=\{1,\ldots,n\}$. Let $e_{i}$ is the $i$th standard basis element for $i\in[n]$. Let $\operatorname{relu}(x)=\max(x,0)$ apply entrywise for $x\in\mathbb{R}^{n}$. Let $\operatorname{diag}(Wx>0)$ be the diagonal matrix that is 1 in the $(i,i)$th entry if $(Wx)_{i}>0$, and 0 otherwise. Let $\mathcal{B}(x,r)$ be the Euclidean ball of radius $r$ centered at $x$. Let $\Pi_{i=d}^{1}W_{i}=W_{d}W_{d-1}\cdots W_{1}$. Let $I_{n}$ be the $n\times n$ identity matrix. Let $A\preceq B$ mean that $B-A$ is a positive semidefinite matrix. For matrices $A$, let $\|A\|$ be the spectral norm of $A$. Let $S^{k-1}$ be the unit sphere in $\mathbb{R}^{k}$. For any nonzero $x\in\mathbb{R}^{n}$, let $\hat{x}=x/\|x\|_{2}$. For a set $S$, let $|S|$ denote its cardinality. We will write $\gamma=O(\delta)$ to mean that there exists a positive constant C such that $\gamma\leq C\delta$, when $\gamma$ is understood to be positive. Similarly we will write $c=\Omega(\delta)$ to mean that there exists a positive constant C such that $c\geq C\delta$. When we say that a constant depends polynomially on $\epsilon^{-1}$, that means that it is at most $C\epsilon^{-k}$ for some positive $C$ and positive integer $k$. Let $\theta_{0}=\angle(x,x_{0})$ and $\overline{\theta}_{1}=g(\theta_{0})$ where $g$ is given by (4). For notational convenience, we will write $a=b+O_{1}(\epsilon)$ if $\|a-b\|\leq\epsilon$, where the norm is understood to be absolute value for scalars, the $\ell_{2}$ norm for vectors, and the spectral norm for matrices. Write $g^{\circ d}$ to denote the composition of $g$ with itself $d$ times. Let $1_{S}=1$ if S and 0 otherwise. For nonzero $v$, let $D_{v}f(x)$ be the (normalized) one-sided directional derivative of $f$ at $x$ in the direction of $v$: $D_{v}f(x)=\lim_{t\to 0^{+}}\frac{f(x+tv)-f(x)}{t\|v\|_{2}}$.

1.3 Main Results

We consider the inverse problem of recovering a vector $y_{0}\in\mathbb{R}^{n}$ from $m\ll n$ linear measurements. To resolve the inherent ambiguity from undersampling, we assume, as a prior, that the vector belongs to the range of a $d$-layer generative neural network $G:\mathbb{R}^{k}\to\mathbb{R}^{n}$, with $k<n$. To recover the vector $y_{0}=G(x_{0})$, we attempt to find the latent code $x_{0}\in\mathbb{R}^{k}$ corresponding to it. We consider a generative network modeled by $G(x)=\operatorname{relu}(W_{d}\ldots\operatorname{relu}(W_{2}\operatorname{relu}(W_{1}x_{0}))\ldots)$, where $\operatorname{relu}(x)=\max(x,0)$ applies entrywise, $W_{i}\in\mathbb{R}^{n_{i}\times n_{i-1}}$, $n_{i}$ is the number of neurons in the $i$th layer, and $k=n_{0}<n_{1}<\cdots<n_{d}=n$. We consider linear measurements of $G(x_{0})$ given by the sampling matrix $A\in\mathbb{R}^{m\times n}$ and consider $k<m\ll n$. The problem at hand is:

[TABLE]

This problem can be viewed in two ways: (1) as above, given compressive measurements of a vector with the prior information that it belongs to the output of a generative neural network, find that vector; or (2), given compressive observations of the output of a generative neural network, find the latent code corresponding to the network’s output by inverting the neural network and compression simultaneously.

As a way to solve the above problem, we consider minimizing the empirical risk objective

[TABLE]

As this objective is nonconvex, there is no a priori guarantee of efficiently finding the global minimum [52]. Approaches such as gradient descent could in principle get stuck in local minima, instead of finding the desired global minimizer $x_{0}$.

In this paper, we consider a fully-connected generative network $G:\mathbb{R}^{k}\to\mathbb{R}^{n}$ with Gaussian weights and no bias term, along with a Gaussian sampling matrix $A\in\mathbb{R}^{m\times n}$. We show that under appropriate conditions and with high probability, $f$ has a strict descent direction everywhere outside two small neighborhoods of $x_{0}$ and a negative multiple of $x_{0}$. We assume that the network is sufficiently expansive at each layer, $n_{i}=\Omega(n_{i-1}\log n_{i-1})$, and that there are a sufficient number of measurements, $m=\Omega(kd\log(n_{1}\cdots n_{d}))$. Let $D_{v}f(x)$ be the (normalized) one-sided directional derivative of $f$ at $x$ in the direction of $v$: $D_{v}f(x)=\lim_{t\to 0^{+}}\frac{f(x+tv)-f(x)}{t\|v\|_{2}}$. Let $\mathcal{B}(x,r)$ be the Euclidean ball of radius $r$ centered at $x$. Our main result is as follows:

Theorem 1**.**

Fix $\epsilon>0$ such that $K_{1}d^{8}\epsilon^{1/4}\leq 1$, and let $d\geq 2$. Assume $n_{i}\geq cn_{i-1}\log n_{i-1}$ for all $i=1\ldots d$ and $m>cdk\log\Pi_{i=1}^{d}n_{i}$. Assume that for each $i$, the entries of $W_{i}$ are i.i.d. $\mathcal{N}(0,1/n_{i})$, and the entries of $A$ are i.i.d. $\mathcal{N}(0,1/m)$ and independent from $\{W_{i}\}$. Then, on an event of probability at least $1-\sum_{i=1}^{d}\tilde{c}n_{i}e^{-\gamma n_{i-1}}-\tilde{c}e^{-\gamma m}$, we have the following. For all nonzero $x$ and $x_{0}$, there exists $v_{x,x_{0}}\in\mathbb{R}^{k}$ such that the one-sided directional derivatives of $f$ satisfy

[TABLE]

where $\rho_{d}$ is a positive number that converges to $1$ as $d\to\infty$. Here, $c$ and $\gamma^{-1}$ are constants that depend polynomially on $\epsilon^{-1}$, and $\tilde{c},K_{1},K_{2}$ are universal constants.

This theorem states that for a network of fixed depth $d$, with high probability there is always a descent direction outside of two specified, sufficiently small neighborhoods, provided that the network is Gaussian and sufficiently expansive. Further, for such networks, zero is a local maximizer. We note that the linear dependence of sample complexity with respect to $k$, for fixed $d$, is optimal. The fixed $d$ regime is realistic to applications because many deep learning networks in the wild have $d$ on the order of only 10. We also note that the theorem’s scalings with respect to $\epsilon$, $d$, and $n_{i}$ are all polynomial, and not exponential, though the dependence on each of these variables could likely be improved. While the sample complexity scaling appears to get worse for larger $d$, we note that larger $d$ allows for the possibility of generative models with lower values of $k$. This is because the number of piecewise linear pieces in $G$ grows exponentially in $d$. Also, note that while the weights of any layer of the network are assumed to be i.i.d. Gaussian, there is no assumption on the independence between $W_{i}$ and $W_{j}$ for $i\neq j$.

The descent direction $v_{x,x_{0}}$ is given by the gradient of $f$:

[TABLE]

where $w$ can be arbitrarily chosen such that $G$ is differentiable at $x+\delta w$ for sufficiently small $\delta$. Such a $w$ exists by the piecewise linearity of $G$, and could be generated randomly with probability 1. An explicit formula for $\nabla f(x)$, where it exists, is given by (5) in Section 2. This expression for $v_{x,x_{0}}$ is in a form that can be computed for any $x$, even for points of nondifferentiability, as part of a gradient based algorithm.

This theorem will be proven by showing the sufficiency of two deterministic conditions on $G$ and $A$, and then by showing that Gaussian $G$ and $A$ of appropriate sizes satisfy these conditions with the appropriate probability. The first deterministic condition is on the spatial arrangement of the network weights within each layer.

Definition 2**.**

We say that the matrix $W\in\mathbb{R}^{n\times k}$ satisfies the Weight Distribution Condition with constant $\epsilon$ if for all nonzero $x,y\in\mathbb{R}^{k}$,

[TABLE]

where $w_{i}\in\mathbb{R}^{k}$ is the $i$th row of $W$; $M_{\hat{x}\leftrightarrow\hat{y}}\in\mathbb{R}^{k\times k}$ is the matrix111A formula for $M_{\hat{x}\leftrightarrow\hat{y}}$ is as follows. If $\theta_{0}=\angle(\hat{x},\hat{y})\in(0,\pi)$ and $R$ is a rotation matrix such that $\hat{x}$ and $\hat{y}$ map to $e_{1}$ and $\cos\theta_{0}\cdot e_{1}+\sin\theta_{0}\cdot e_{2}$ respectively, then $M_{\hat{x}\leftrightarrow\hat{y}}=R^{t}\begin{pmatrix}\cos\theta_{0}&\sin\theta_{0}&0\\ \sin\theta_{0}&-\cos\theta_{0}&0\\ 0&0&0_{k-2}\end{pmatrix}R$, where $0_{k-2}$ is a $k-2\times k-2$ matrix of zeros. If $\theta_{0}=0$ or $\pi$, then $M_{\hat{x}\leftrightarrow\hat{y}}=\hat{x}\hat{x}^{t}$ or $-\hat{x}\hat{x}^{t}$, respectively. such that $\hat{x}\mapsto\hat{y}$, $\hat{y}\mapsto\hat{x}$, and $z\mapsto 0$ for all $z\in\operatorname{span}(\{x,y\})^{\perp}$; $\hat{x}=x/\|x\|_{2}$ and $\hat{y}=y/\|y\|_{2}$; $\theta_{0}=\angle(x,y)$; and $1_{S}$ is the indicator function on $S$.

The norm on the left hand side of (2) is the spectral norm. Note that an elementary calculation222To do this calculation, take $x=e_{1}$ and $y=\cos\theta_{0}\cdot e_{1}+\sin\theta_{0}\cdot e_{2}$ without loss of generality. Then each entry of the matrix can be determined analytically by an integral that factors in polar coordinates. gives that $Q_{x,y}=\mathbb{E}[\sum_{i=1}^{n}1_{w_{i}\cdot x>0}1_{w_{i}\cdot y>0}\cdot w_{i}w_{i}^{t}]$ for $w_{i}\sim\mathcal{N}(0,I_{k}/n)$. As the rows $w_{i}$ correspond to the neural network weights of the $i$th neuron in a layer given by $W$, the WDC provides a deterministic property under which the set of neuron weights within the layer given by $W$ are distributed approximately like a Gaussian. The WDC could also be interpreted as a deterministic property under which the neuron weights are distributed approximately like a uniform random variable on a sphere of a particular radius. Note that if $x=y$, $Q_{x,y}$ is an isometry up to a factor of $1/2$.

The second deterministic condition is that the compression matrix acts like an isometry on pairs of differences of vectors in the range of $G:\mathbb{R}^{k}\to\mathbb{R}^{n}$.

Definition 3**.**

We say that the compression matrix $A\in\mathbb{R}^{m\times n}$ satisfies the Range Restricted Isometry Condition (RRIC) with respect to $G$ with constant $\epsilon$ if for all $x_{1},x_{2},x_{3},x_{4}\in\mathbb{R}^{k}$,

[TABLE]

We can now state our main deterministic result.

Theorem 4**.**

Fix $\epsilon>0$ such that $K_{1}d^{8}\epsilon^{1/4}\leq 1$, and let $d\geq 2$. Suppose that $G$ is such that $W_{i}$ has the WDC with constant $\epsilon$ for all $i=1\ldots d$. Suppose $A$ satisfies the RRIC with respect to $G$ with constant $\epsilon$. Then, for all nonzero $x$ and $x_{0}$, there exists $v_{x,x_{0}}\in\mathbb{R}^{k}$ such that the one-sided directional derivatives of $f$ satisfy

[TABLE]

where $\rho_{d}$ is a positive number that converges to $1$ as $d\to\infty$, and $K_{1}$, $K_{2}$, and $K_{3}$ are universal constants.

Note that the $2^{d}$ scaling in the bounds is an artifact of the scaling of the problem and does not indicate a vanishingly small derivative. Roughly speaking, the $\operatorname{relu}$ activation functions zero out roughly half of its arguments. Hence, while $W_{i}$ has spectral norm approximately $1$, the rows of $W_{i}$ that are retained by the $\operatorname{relu}$ will have spectral norm approximately $1/2$. Thus, $f(x)$ itself is on the order of $2^{-d}$ under the RRIC and WDC for appropriately small $\epsilon$.

In the case that $A=I_{n}$, the RRIC is trivially satisfied, and we get the following corollary about inverting multilayer neural networks.

Corollary 5** (Approximate Invertibility of Multilayer Neural Networks).**

If $G$ is a $d$-layer neural network such that $W_{i}$ satisfies the WDC with constant $\epsilon$ for all $i=1\ldots d$, then the function $f(x)=\|G(x)-G(x_{0})\|_{2}$ has no stationary points outside of a neighborhood around $x_{0}$ and $-\rho_{d}x_{0}$.

In the case of a Gaussian network with Gaussian measurements, the WDC and RRIC are satisfied with high probability if the network is sufficiently expansive and there are a sufficient number of measurements.

Proposition 6**.**

Fix $0<\epsilon<1$. Assume $n_{i}\geq cn_{i-1}\log n_{i-1}$ for all $i=1\ldots d$ and $m>cdk\log\Pi_{i=1}^{d}n_{i}$. Assume the entires of $W_{i}$ are i.i.d. $\mathcal{N}(0,1/n_{i})$, and the entries of $A$ are i.i.d. $\mathcal{N}(0,1/m)$. Then, $W_{i}$ satisfies the WDC with constant $\epsilon$ for all $i$ and $A$ satisfies the RRIC with respect to $G$ with constant $\epsilon$ with probability at least $1-\sum_{i=1}^{d}\tilde{c}n_{i}e^{-\gamma n_{i-1}}-\tilde{c}e^{-\gamma m}$. Here, $c$ and $\gamma^{-1}$ are constants that depend polynomially on $\epsilon^{-1}$, and $\tilde{c}$ is a universal constant.

As stated after Theorem 1, no assumption is made on the independence between $W_{i}$ and $W_{j}$ for $i\neq j$. While Proposition 6 is stated for $A\in\mathbb{R}^{m\times n}$ with i.i.d. Gaussian entries, it also applies in the case of any random matrix that satisfies the following concentration of measure condition:

[TABLE]

for any fixed $x\in\mathbb{R}^{n}$, where $c_{0}(\epsilon)$ is a positive constant depending only on $\epsilon$. In particular, Proposition 6 and hence Theorem 1 extends to the case of where the entries of $A$ are independent Bernoulli random variables (and the entries of $W_{i}$ are Gaussian). See [7] for more.

1.4 Discussion

In this paper, we provide the first rigorous global analysis of the efficacy of enforcing generative neural network priors. We show that if a generative neural network has Gaussian weights and is sufficiently expansive at each layer, then, with high probability, the empirical risk objective applied to the network output has no spurious local minima or saddle points outside two small neighborhoods around the global optimum and a negative reflection of it. Further, if the output of the network is subject to random Gaussian compressive measurements, then the same conclusion holds with information theoretically optimal sample complexity with respect to the latent code dimensionality. That is, a convergent gradient descent scheme will approximately invert the generative network, even in the presence of a sufficient number of compressive measurements.

As this theoretical work is the first of its kind, it leaves open many important questions deserving further research. Because any particular generative network is unlikely to contain an observed image exactly, it is important to establish a similar guarantee to Theorem 1 in the case that the observed image is not in the range of the generative network. This line of work includes establishing noise tolerance and robustness to outliers, both of which have been established for sparsity-based compressed sensing. Such results would provide even further theoretical support for several empirical observations about enforcing generative priors via an optimization over latent code space [8, 73], including empirical robustness of inverting generative models [36]. In particular, it could help explain the significant observation that generative priors can mitigate against adversarial examples [27, 61], which are minor and sometimes imperceptible modifications to images that lead to catastrophic misclassification by neural networks [64]. Robustness against adversarial examples is important for the security of machine learning systems [72, 3], in particular those that will be part of self-driving cars.

In this work, we assume that weights of the generative network are modeled by Gaussians. There is empirical evidence justifying this assumption for trained neural networks [4]. Additionally, previous theoretical work with neural networks, in the area of classification, has also studied Gaussian networks [21]. As was the case with compressed sensing, where theoretical developments with Gaussians inspired subsequent theory with more realistic measurement models, the novel results of this paper motivate additional theoretical with more complicated assumptions on the weights of generative networks. Further, while the generative model assumed in this paper captures key structural elements of real neural networks (each layer acting as a nonlinear function of a linear transformation), this work also motivates establishing similar results for more complicated network structures, including bias terms, convolutional layers, max-pooling, and more.

Most importantly, we provide in this paper a theoretical framework for studying the enforcement of deep generative priors via empirical risk as a means of regularization on inverse problems. Besides compressive sensing with linear measurements, there are a myriad of inverse problems that may benefit from such an approach. One particularly exciting example is the field of phase retrieval, which is critical in the biological sciences for X-ray crystallography and modern techniques like XFEL-imaging, which is a promising approach that may lead to breakthroughs in understanding of proteins and other molecular structures. Phase retrieval involves recovering vectors from quadratic observations, and enforcing linear sparsity priors subject to such quadratic measurements has been met with potentially fundamental limitations of polynomial time algorithms. In particular, while the theoretically optimal sample complexity of sparse phase retrieval is O({\color[rgb]{0,0,0}s}\log n), where $s$ is the sparsity of the signal, it is potentially unobtainable via polynomial time algorithms [6], which have so far only produced O({\color[rgb]{0,0,0}s}^{2}\log n) efficient reconstruction schemes [35]. This bottleneck in sample complexity makes improvements in signal priors critical for the field of phase retrieval to advance. The present work indicates that it may be possible to use generative priors for problems such as phase retrieval. If a bound similar to that in our Theorem is true in phase retrieval, it would mean that recovery could be possible with $O(k)$ measurements, where $k$ is the latent code dimensionality. This could beat sparsity based approaches both because the scaling is linear in the signal’s latent dimensionality of the representation, and because $k$ can be smaller than $s$ for the very same signal.

More significantly, the regime of using generative modeling as a means of regularization opens new doors for improving the workflow of biological scientists. In modern phase retrieval, modeling assumptions which aid in lowering sample complexity and increasing SNR are all hand-coded, making the process extremely tedious. In contrast, deep generative modeling simply requires obtaining a dataset of previously reconstructed molecular structures, which are easily available in extensive databases amassed over the years of practice of crystallography [65]. One may envision training generative models on such datasets and using the resulting neural network priors to regularize the inverse problem of phase retrieval, tabula rasa, and potentially more effectively than hand-modeling ever could, as has been witnessed in the field of computer vision. This makes possible recovering the structure of biological molecules without explicit modeling, freeing up scientists to focus on innovating on new imaging modalities instead of grappling with the tedium of hand-coding their prior knowledge to solve the resulting inverse problems. More broadly, combining the power of deep generative modeling with modern methods of optimization and signal recovery, allows potentially paradigm shifting improvements to the empirical sciences, by taking a data-driven artificial intelligence approach to signal recovery.

2 Proofs

The theorems are proven by a concentration argument. We show that $v_{x,x_{0}}\in\mathbb{R}^{k}$ concentrates around a particular $h_{x,x_{0}}\in\mathbb{R}^{k}$ that is a continuous function of nonzero $x,x_{0}$ and is zero only at $x=x_{0}$ and $x=-\rho_{d}x_{0}$. Before we sketch the proof below, we introduce some useful quantities.

In order to analyze which rows of a matrix $W$ are active when computing $\operatorname{relu}(Wx)$, we let

[TABLE]

For a fixed $W$, the matrix $W_{+,x}$ zeros out the rows of $W$ that do not have a positive dot product with $x$. Alternatively put, $W_{+,x}$ contains weights from only the neurons that are active for the input $x$. We also define $W_{1,+,x}=(W_{1})_{+,x}=\operatorname{diag}(W_{1}x>0)W_{1}$ and

[TABLE]

The matrix $W_{i,+,x}$ consists only of the neurons in the $i$th layer that are active if the input to the first layer is $x$. Additionally, it will be useful to control how the operator $x\mapsto W_{+,x}x$ distorts angles. In order to study this, we define

[TABLE]

We now specify the choice of $v_{x,x_{0}}$ as follows. At any $x\in\mathbb{R}^{k}$ such that $G$ is differentiable at $x$,

[TABLE]

Let $w\in\mathbb{R}^{k}$ be such that $G$ is differentiable at $x+\delta w$ for sufficiently small $\delta$. Such a $w$ exists by the piecewise linearity of $G$. Let

[TABLE]

Note that the first part of the definition of $v_{x,x_{0}}$ can be viewed as the special case of the second part of the definition with $w=0$. When $G$ is not differentiable at $x$, multiple values of $v_{x,x_{0}}$ are consistent with the above definition. These values correspond to the multiple choices of $w$. The concentration analysis applies simultaneously for all appropriate $w$ because of uniformity in the concentration results below.

A sketch of the proof is as follows:

• •

The WDC and RRIC imply that

[TABLE]

uniformly over nonzero $x$ and $x_{0}$. See the proof of Theorem 4 in Section 2.3.

• •

The WDC implies that

[TABLE]

uniformly over nonzero $x$ and $x_{0}$, where $\overline{\theta}_{i}=g(\overline{\theta}_{i-1})$, $\overline{\theta}_{0}=\angle(x,x_{0})$, and $h_{x,x_{0}}$ is continuous for nonzero $x,x_{0}$. See Sections 2.1 and 2.2.

• •

Direct analysis shows that $h_{x,x_{0}}\approx 0$ only within a neighborhood of $x_{0}$ and $-\rho_{d}x_{0}$. See Section 2.4.

• •

Arguments from probabilistic concentration theory establish that the WDC and RRIC with high probability for Gaussian matrices of appropriate dimensions. See Sections 2.5 and 2.6, respectively. These together establish Proposition 6.

Theorem 1 is the combination of Theorem 4 and Proposition 6.

The proof capitalizes on the structure of the $\operatorname{relu}$ nonlinearities because it considers points in the range of the generator $G$ to lie in the union of finitely many subspaces, each given by the range of all possible matrices $W_{i,+,x}$. Probabilistic concentration of these matrices requires a bound on the maximum number of such subspaces. These bounds would be worse or possibly infinite for nonlinearities other than $\operatorname{relu}$. While the nondeterministic result is stated for Gaussian weight matrices $W_{i}$, the same analysis would extend to weight matrices whose rows are given by a uniform distribution over a sphere of appropriate radius, as the proof capitalizes on rotational invariance of the neuronal weights.

2.1 Approximate angle contraction property $W_{+,x}$

In the concentration result of the next section, we will make use of the fact that the angle between $W_{+,x}x$ and $W_{+,y}y$ is approximately g\bigl{(}\angle(x,y)\bigr{)} if the WDC holds. As Figure 1 shows, $g$ is monotonic and less than the identity. Thus, the mapping $x\mapsto W_{+,x}x$ has an approximate angle contraction property in the sense of the following lemma.

Lemma 7**.**

Fix $0<\epsilon<0.1$. Let $W\in\mathbb{R}^{n\times k}$ satisfy the WDC with constant $\epsilon$. We have $\theta_{1}:=\angle(W_{+,x}x,W_{+,y}y)$ is well-defined for all $x\neq 0,y\neq 0$, and

[TABLE]

where $\theta_{0}=\angle(x,y)$ and $g$ is defined by (4).

Proof.

It suffices to establish that for all $x\neq 0,y\neq 0$,

[TABLE]

Without loss of generality, consider only $x,y\in S^{k-1}$. By the WDC, $\|W_{+,x}^{t}W_{+,y}-Q_{x,y}\|\leq\epsilon$ for all $x,y\in S^{k-1}$. Let

[TABLE]

We have $\max(|\delta_{1}|,|\delta_{2}|,|\delta_{3}|)\leq\epsilon$ for all $x,y\in S^{k-1}$. As $|\delta_{2}|\leq\epsilon$, |\delta_{3}|\leq{\color[rgb]{0,0,0}\epsilon}, and $\epsilon<1/2$, $W_{+,x}x\neq 0$ and $W_{+,y}y\neq 0$. Thus, $\theta_{1}$ is well-defined. We have

[TABLE]

Thus,

[TABLE]

where the second line follows as $2|\langle x,Q_{x,y}y\rangle|\leq 2\|Q_{x,y}\|\leq 1$ and $\max(|\delta_{1}|,|\delta_{2}|,|\delta_{3}|)\leq\epsilon$, and the last line follows because $\epsilon<0.1$. The proof is concluded by noting that 2\langle x,Q_{x,y}y\rangle=\frac{1}{\pi}\bigl{[}(\pi-\theta_{0})\cos\theta_{0}+\sin\theta_{0}\bigr{]}.

∎

2.2 Concentration of terms without compression

At points of differentiability $x$, we have

[TABLE]

In this section, we prove that the WDC establishes concentration for the terms in $v_{x,x_{0}}$ uniformly in $x$ and $x_{0}$ in the compressionless case of $A=I_{n}$. The case with compression will use the concentration of these terms too.

Lemma 8**.**

Fix $0<\epsilon<d^{-4}/(16\pi)^{2}$ and $d\geq 2$. Suppose that $W_{i}\in\mathbb{R}^{n_{i}\times n_{i-1}}$ satisfies the WDC with constant $\epsilon$ for $i=1\ldots d$. Define

[TABLE]

where $\bar{\theta}_{i}=g(\overline{\theta}_{i-1})$ for $g$ given by (4) and $\overline{\theta}_{0}=\angle(x,y)$. For all $x\neq 0$ and $y\neq 0$,

[TABLE]

Proof.

Part I: Assembling some useful bounds.

Define $x_{0}=x$, $y_{0}=y$,

[TABLE]

and analogously $y_{d}=\left(\Pi_{i=d}^{1}W_{i,+,y}\right)y$, where $(W_{i})_{+,x}=\operatorname{diag}(W_{i}x>0)W_{i}$.

By the WDC, we have for all $i=1\ldots d$, for all $x\neq 0,y\neq 0$,

[TABLE]

In particular, $\|W_{i,+,x}^{t}W_{i,+,y}-Q_{x_{i-1},y_{i-1}}\|\leq\epsilon$. We now detail several bounds that follow from the WDC. Most immediately, we have that for all $x\neq 0$ and for all $i=1\ldots d$,

[TABLE]

and consequently,

[TABLE]

Hence,

[TABLE]

where we used that $\log(1+z)\leq z$, $e^{z}\leq 1+2z$ for $z<1$, and $2d\epsilon<1$. We also have for all $x\neq 0$ that

[TABLE]

Thus, we have that for all $x,y\neq 0$,

[TABLE]

Note that if $4\epsilon d<1$,

[TABLE]

where we used that $\frac{1+z}{1-z}\leq 1+4z$ for $0\leq z\leq\frac{1}{2}$, $e^{z}\leq 1+2z$ for $0\leq z\leq 1$, and $4d\epsilon<1$. As $1-z\leq(1+z)^{-1}$ for $z>0$, we also have

[TABLE]

and thus, if $4\epsilon d<1$,

[TABLE]

By Lemma 7, the WDC implies that $\theta_{V_{+,x}x,V_{+,y}y}$ is well-defined for all nonzero $x$ and $y$, and for $V=W_{1},\ldots,W_{d}$, and

[TABLE]

where $g(\theta)$ is given by (4), and $\delta:=4\sqrt{\epsilon}$. Define $\theta_{i}:=\angle(x_{i},y_{i})\in[0,\pi]$. Note that by (14), we have333See Section 1.2 for the meaning of $O_{1}$. $\theta_{d}=g(\theta_{d-1})+O_{1}(\delta)$ for all $d$, and thus $\theta_{d}=g(g(\cdots g(g(\theta_{0})+O_{1}(\delta))+O_{1}(\delta)\cdots)+O_{1}(\delta))+O_{1}(\delta)$. Because $|g^{\prime}(\theta)|\leq 1$ for all $\theta$ and because $\overline{\theta}_{d}=g(g(\cdots g(\theta_{0})\cdots))=g^{\circ d}(\theta_{0})$, we have

[TABLE]

*Part II: Establishing (8)

*We have that $\cos\theta_{d}\geq 3/(4\pi)$ by combining (15), $\overline{\theta}_{d}\leq\cos^{-1}(\frac{1}{\pi})$ for $d\geq 2$, and $4\pi d4\sqrt{\epsilon}\leq 1$. Additionally, by (12), we have \|x_{d}\|_{2}\|y_{d}\|_{2}\geq\|x\|_{2}\|y\|_{2}\bigl{(}\frac{1}{2}-\epsilon\bigr{)}^{d}\geq\|x\|_{2}\|y\|_{2}\frac{1-2d\epsilon}{2^{d}}, where the last inequality holds as $\epsilon\leq 1/2$. If $2d\epsilon\leq 2/3$, we have

[TABLE]

which establishes (8).

*Part III: Establishing (7)

*This proof will proceed by setting up and solving a recurrence relation. We will use that

[TABLE]

First, we derive a recurrence relation and solve for $(\Pi_{i=d}^{1}W_{i,+,x})^{t}(\Pi_{i=d}^{1}W_{i,+,x})$. We have

[TABLE]

where the first equality follows by (10), and the third equality follows from (11), as $2\epsilon d\leq 1$. Solving this recurrence relation with $M_{0}=I_{n_{0}}$ by (16), we get that if $4d\epsilon\leq 1$, then

[TABLE]

Thus,

[TABLE]

Next, we derive a recurrence relation for $\Gamma_{d}:=(\Pi_{i=d}^{1}W_{i,+,x})^{t}(\Pi_{i=d}^{1}W_{i,+,y})y$.

[TABLE]

where the second line follows from (9), the definition $W_{d,+,y}=(W_{d})_{+,y_{d-1}}$, and the definition of $Q_{x_{d},y_{d}}$ in (2); the third line follows by the definition of $M_{\hat{x}_{d-1}\leftrightarrow\hat{y}_{d-1}}$ in (2), the bound (11), and the definition of $x_{d-1}$; the fourth line uses (17); the fifth line follows from $4\epsilon d\leq 1$ and (13); the eighth line follows from (13); and the last line follows from (15), $\delta=4\sqrt{\epsilon}$, and $11\sqrt{\epsilon}\leq 4$.

Solving the recurrence relation (20) using (16), we get that

[TABLE]

First, we control the first term of $\Gamma_{d}$ in (21). We have

[TABLE]

where the first inequality follows by Lemma 9 and as $\frac{\pi-\overline{\theta}_{i}}{2\pi}\in[0,1/2]$; and fifth inequality follows because $e^{d^{2}\delta/\pi}\leq 1+2\frac{d^{2}\delta}{\pi}$ if $d^{2}\delta/\pi\leq 1$.

Next, we control the second term of $\Gamma_{d}$ in (21). Similar to the calculation above, we have

[TABLE]

if $d^{2}\delta/\pi\leq 1$. We now get that the second term of (21) is

[TABLE]

where the first equality follows by (23) and $d^{2}\delta/\pi\leq 1$, and the second equality follows by expanding the terms and using $d^{2}\delta\leq 1$.

Combining (22) and (24), we get

[TABLE]

We complete the proof of (7) by combining this equality with (18), and $\delta=4\sqrt{\epsilon}$.

∎

In the proof of the previous lemma, we used the following technical result.

Lemma 9**.**

Let $d\in\mathbb{N}$ and let $0\leq r_{i}\leq r_{\text{max}}$ for $i=1\ldots d$. We have

[TABLE]

Proof.

First, we establish that

[TABLE]

This follows by noting that for $\delta\in[0,1]$,

[TABLE]

and integrating over $\delta\in[0,1]$. Next, we establish that

[TABLE]

This follows by noting that for all $k=1\ldots d$,

[TABLE]

∎

2.3 Proof of Deterministic Theorem

The proof of Theorem 4 follows the outline provided in Section 2.

Proof of Theorem 4.

Recall that

[TABLE]

where $G$ is differentiable at $x+\delta w$ for sufficiently small $\delta$. Such a $w$ exists by the piecewise linearity of $G$, and any such $w$ can be selected arbitrarily. Also, recall that

[TABLE]

Let

[TABLE]

where $\overline{\theta}_{i}=g(\overline{\theta}_{i-1})$ and $\overline{\theta}_{0}=\angle(x,x_{0})$. For brevity of notation, write $v_{x}=v_{x,x_{0}},\overline{v}_{x}=\overline{v}_{x,x_{0}},$ and $h_{x}=h_{x,x_{0}}$.

The WDC implies that for all $x\neq 0$ and for all $i=1\ldots d$,

[TABLE]

Now, we establish that for all differentiable points $x\in\mathbb{R}^{k}$,

[TABLE]

At $x\in\mathbb{R}^{k}$ such that $G$ is differentiable at $x$, the local linearity of $G$ gives that $G(x+z)-G(x)=(\Pi_{i=d}^{1}W_{i,+,x})z$ for any sufficiently small $z\in\mathbb{R}^{k}$. By the RRIC, we have

[TABLE]

for all $z,\tilde{z}$, which, together with (25), implies (26).

Similarly, the WDC implies by Lemma 8 and $\epsilon<1/(16\pi d^{2})^{2}$ that for all nonzero $x,x_{0},y\in\mathbb{R}^{k}$,

[TABLE]

Thus, we have, for all $x\neq 0,x_{0}\neq 0$,

[TABLE]

for some universal constant $\tilde{K}$, where the first inequality follows by the definition of $v_{x}$ and the continuity of $h_{x}$ for nonzero $x$; the second inequality follows by combining (25), (26), (28), and as $2d\epsilon\leq 1\Rightarrow(1+2\epsilon)^{d}\leq e^{2\epsilon d}\leq 1+4\epsilon d$.

Note that the one-sided directional derivative of $f$ in the direction of $y\neq 0$ at $x$ is $D_{y}f(x)=\lim_{t\to 0^{+}}\frac{f(x+ty)-f(x)}{t}$. Due to the continuity and piecewise linearity of the function

[TABLE]

we have that for any $x,y\neq 0$ that there exists a sequence $\{x_{n}\}\to x$ such that $f$ is differentiable at each $x_{n}$ and $D_{y}f(x)=\lim_{n\to\infty}\nabla f(x_{n})\cdot y$. Thus, as $\nabla f(x_{n})=v_{x_{n}}$,

[TABLE]

Now, we write

[TABLE]

where the first inequality follows from the triangle inequality, and the second inequality follows by (30). As $h_{x}$ is continuous in $x$ for all nonzero $x$, we have for any x\in S_{{\color[rgb]{0,0,0}8\sqrt{\epsilon}}\tilde{K}d^{3}}^{c},

[TABLE]

where the last inequality uses (30) and the definition of S_{{\color[rgb]{0,0,0}8\sqrt{\epsilon}}\tilde{K}d^{3}}. We conclude $D_{-v_{x}}f(x)<0$ for all nonzero x\in S_{{\color[rgb]{0,0,0}8}\sqrt{\epsilon}\tilde{K}d^{3}}^{c}.

It remains to prove that $\forall x\neq 0$, $D_{x}f(0)<0$. We compute that

[TABLE]

where the first inequality holds by (27), (25), and (29); and the second inequality follows from $4\epsilon d\leq 1$. Thus, for \epsilon<{\color[rgb]{0,0,0}\frac{1}{16\pi}}, D_{x}f(0)<{\color[rgb]{0,0,0}-\frac{1}{8\pi 2^{d}}\|x_{0}\|_{2}}.

The proof is finished by applying Lemma 10 and 8\pi d^{6}\sqrt{{\color[rgb]{0,0,0}8}\sqrt{\epsilon}\tilde{K}d^{3}}\leq 1 to get

[TABLE]

∎

2.4 Control of the zeros of $h_{x,x_{0}}$

We now show that $h_{x,x_{0}}$ is away from zero outside of a neighborhood of $x_{0}$ and $-\rho_{d}x_{0}$.

Lemma 10**.**

Suppose $8\pi d^{6}\sqrt{\epsilon}\leq 1$. Let

[TABLE]

where $d$ is an integer greater than $1$ and let $h_{x,x_{0}}$ be defined by

[TABLE]

where $\overline{\theta}_{0}=\angle(x,x_{0})$ and $\overline{\theta}_{i}=g(\overline{\theta}_{i-1})$ for $g$ given by (4). Define

[TABLE]

where $\widecheck{\theta}_{0}=\pi$ and $\widecheck{\theta}_{i}=g(\widecheck{\theta}_{i-1})$. If $x\in S_{\epsilon,x_{0}}$, then we have that either

[TABLE]

or

[TABLE]

In particular, we have

[TABLE]

Additionally, $\rho_{d}\to 1$ as $d\to\infty$.

Proof.

Without loss of generality, let $\|x_{0}\|_{2}=1$, $x_{0}=e_{1}$ and {\color[rgb]{0,0,0}x}=r\cos\overline{\theta}_{0}\cdot e_{1}+r\sin\overline{\theta}_{0}\cdot e_{2} for $\overline{\theta}_{0}\in[0,\pi]$. Let $x\in S_{\epsilon,x_{0}}$.

First we introduce some notation for convenience. Let

[TABLE]

Thus, h_{x,x_{0}}=-{\color[rgb]{0,0,0}\frac{1}{2^{d}}}\xi\hat{x}_{0}+{\color[rgb]{0,0,0}\frac{1}{2^{d}}}(r-\zeta)\hat{x}. By inspecting the components of {\color[rgb]{0,0,0}h_{x,x_{0}}}, we have that $x\in S_{\epsilon,x_{0}}$ implies

[TABLE]

Now, we record several properties. We have:

[TABLE]

We now establish (37). Observe 0<g(\theta)\leq\bigl{(}\frac{1}{3\pi}+\frac{1}{\theta}\bigr{)}^{-1}=:\tilde{g}(\theta) for $\theta\in(0,\pi]$. As $g$ and $\tilde{g}$ are monotonic increasing, we have \widecheck{\theta}_{i}=g^{\circ i}(\widecheck{\theta}_{0})=g^{\circ i}(\pi)\leq\tilde{g}^{\circ i}(\pi)=\bigl{(}\frac{i}{3\pi}+\frac{1}{\pi}\bigr{)}^{-1}=\frac{3\pi}{i+3}. Similarly, $g(\theta)\geq(\frac{1}{\pi}+\frac{1}{\theta})^{-1}$ implies that $\widecheck{\theta}_{i}\geq\frac{\pi}{i+1}$, establishing (38).

We now establish (39). Using (37) and $\overline{\theta}_{i}\leq\widecheck{\theta}_{i}$, we have

[TABLE]

where the last inequality can be established by showing that the ratio of consecutive terms with respect to $d$ is greater for the product in the middle expression than for $d^{-3}$.

We establish (40) by using the fact that $|g^{\prime}(\theta)|\leq 1$ for all $\theta\in[0,\pi]$ and using the same logic as for (15).

We now establish (42). As $\overline{\theta}_{0}=\pi+O_{1}(\delta)$, we have $\overline{\theta}_{i}=\widecheck{\theta}_{i}+O_{1}(i\delta)$. Thus, if $\frac{d^{2}\delta}{\pi}\leq 1$,

[TABLE]

So

[TABLE]

Thus (42) holds.

Next, we establish that $x\in S_{\epsilon,x_{0}}\Rightarrow r\leq 4d$, and thus $M\leq 4d$. Suppose $r>1$. At least one of the following holds: $|\sin\overline{\theta}_{0}|\geq 1/\sqrt{2}$ or $|\cos\overline{\theta}_{0}|\geq 1/\sqrt{2}$. If $|\sin\overline{\theta}_{0}|\geq 1/\sqrt{2}$ then (34) implies that $|r-\zeta|\leq\sqrt{2}\epsilon r$. Using (36), we get $r\leq\frac{d/\pi}{1-\sqrt{2}\epsilon}\leq d/2$ if $\epsilon<1/4$. If $|\cos\overline{\theta}_{0}|\geq 1/\sqrt{2}$, then (33) implies that $|r-\zeta|\leq\sqrt{2}(\epsilon r+|\xi|)$. Using (35), (36), and $\epsilon<1/4$, we get $r\leq\frac{\sqrt{2}|\xi|+\zeta}{1-\sqrt{2}{\epsilon}}\leq\frac{\sqrt{2}+d}{1-\sqrt{2}\epsilon}\leq 4d$. Thus, we have $x\in S_{\epsilon}\Rightarrow r\leq 4d\Rightarrow M\leq 4d$.

Next, we establish that we only need to consider the small angle case ($\overline{\theta}_{0}\approx 0$) and the large angle case ($\overline{\theta}_{0}\approx\pi$). Exactly one of the following holds: $|r-\zeta|\geq\sqrt{\epsilon}M$ or $|r-\zeta|<\sqrt{\epsilon}M$. If $|r-\zeta|\geq\sqrt{\epsilon}M$, then by (34), we have $|\sin\overline{\theta}_{0}|\leq\sqrt{\epsilon}$. Hence $\overline{\theta}_{0}=O_{1}(2\sqrt{\epsilon})$ or $\overline{\theta}_{0}=\pi+O_{1}(2\sqrt{\epsilon})$, as $\epsilon<1$. If $|r-\zeta|\leq\sqrt{\epsilon}M$, then by (33) we have $|\xi|\leq 2\sqrt{\epsilon}M$. Using (39), we get $\overline{\theta}_{0}=\pi+O_{1}(2\pi d^{3}\sqrt{\epsilon}M)$. Thus, we only need to consider the small angle case, $\overline{\theta}_{0}=O_{1}(2\sqrt{\epsilon})$ and the large angle case $\overline{\theta}_{0}=\pi+O_{1}(8\pi d^{4}\sqrt{\epsilon})$, where we have used $M\leq 4d$.

Small Angle Case. Assume $\overline{\theta}_{0}=O_{1}(2\sqrt{\epsilon})$. As $\overline{\theta}_{i}\leq\overline{\theta}_{0}\leq 2\sqrt{\epsilon}$ for all $i$, we have $\xi\geq(1-\frac{2\sqrt{\epsilon}}{\pi})^{d}=1+O_{1}(\frac{4d\sqrt{\epsilon}}{\pi})$ provided $2d\sqrt{\epsilon}\leq 1/2$. By (36), we also have $\zeta=O_{1}(\frac{d}{\pi}2\sqrt{\epsilon})=O_{1}(d\sqrt{\epsilon})$. By (33), we have

[TABLE]

Thus, as $\cos\overline{\theta}_{0}=1+O_{1}(\overline{\theta}_{0}^{2}/2)=1+O_{1}(2\epsilon)$,

[TABLE]

and thus,

[TABLE]

Large Angle Case. Assume $\theta_{0}=\pi+O_{1}(\delta)$ where $\delta=8\pi d^{4}\sqrt{\epsilon}$. By (41) and (42), we have $\xi=O_{1}(\delta/\pi)$, and we have $\zeta=\rho_{d}+O_{1}(3d^{3}\delta)$ if $8d^{6}\sqrt{\epsilon}\leq 1$. By (33), we have

[TABLE]

so, as $\cos\theta_{0}=1-O_{1}(\theta_{0}^{2}/2)$,

[TABLE]

and thus, using $r\leq 4d$, $\rho_{d}\leq d$, and $\delta=8\pi d^{4}\sqrt{\epsilon}\leq 1$,

[TABLE]

To conclude the proof of (32), we use the fact that

[TABLE]

This fact simply says that if a 2d point is known to have magnitude within $\Delta r$ of some $r$ and is known to be within angle $\Delta\theta$ from [math], then its Euclidean distance to the point of polar coordinates $(r,0)$ is no more than $\Delta r+(r+\Delta r)\Delta\theta$.

Finally, we establish that $\rho_{d}\to 1$ as $d\to\infty$. Note that $\rho_{d+1}=(1-\frac{\widecheck{\theta}_{d}}{\pi})\rho_{d}+\frac{\sin\widecheck{\theta}_{d}}{\pi}$ and $\rho_{0}=0$. It suffices to show $\tilde{\rho}_{d}\to 0$, where $\tilde{\rho}_{d}:=1-\rho_{d}$. The following recurrence relation holds: $\tilde{\rho}_{d}=(1-\frac{\widecheck{\theta}_{d-1}}{\pi})\tilde{\rho}_{d-1}+\frac{\widecheck{\theta}_{d-1}-\sin\widecheck{\theta}_{d-1}}{\pi}$, with $\tilde{\rho}_{0}=1$. Using the recurrence formula (16) and the fact that $\widecheck{\theta}_{0}=\pi$, we get that

[TABLE]

using (38), we have that

[TABLE]

Using (37) and the fact that \widecheck{\theta}_{{\color[rgb]{0,0,0}i}-1}-\sin\widecheck{\theta}_{{\color[rgb]{0,0,0}i}-1}\leq\widecheck{\theta}_{{\color[rgb]{0,0,0}i}-1}^{3}/6, we have that \tilde{\rho}_{d}\leq\sum_{i=1}^{d}\frac{\widecheck{\theta}_{{\color[rgb]{0,0,0}i}-1}^{3}}{6\pi}\cdot\frac{i+{\color[rgb]{0,0,0}1}}{d+{\color[rgb]{0,0,0}1}}\to 0 as $d\to\infty$.

∎

2.5 Proof of WDC for Gaussian Matrices

In this section, we establish a bound on the probability that a Gaussian matrix satisfies the WDC, provided it corresponds to a sufficiently expansive layer of a neural network.

Lemma 11**.**

Fix $0<\epsilon<1$. Let $W\in\mathbb{R}^{n\times k}$ have i.i.d. $\mathcal{N}(0,1/n)$ entries. If $n>ck\log k$, then with probability at least $1-8ne^{-\gamma k}$, $W$ satisfies the WDC with constant $\epsilon$. Here $c,\gamma^{-1}$ are constants that depend only polynomially on $\epsilon^{-1}$.

The WDC with constant $\epsilon$ can be written as

[TABLE]

for all nonzero $x,y\in\mathbb{R}^{k}$. A common way to establish concentration of a random function simultaneously over an infinite number of values of $(x,y)$, like (52), is as follows:

• •

Show concentration of the quantity with high probability for a fixed $(x,y)$.

• •

Bound the Lipschitz constant of the quantity with respect to $(x,y)$.

• •

Take a union bound over a net whose size is given by the Lipschitz constant.

This argument does not apply in the present case because $W_{+,x}^{t}W_{+,y}$ is not continuous with respect to $(x,y)$. To deal with this lack of continuity, we form two continuous variants that are greater and less than $W_{+,x}^{t}W_{+,y}$, respectively, with respect the semidefinite ordering. We now introduce some notation in order to state these bounds. Let

[TABLE]

When applied to a vector, let $h_{-\epsilon}$ and $h_{\epsilon}$ act component-wise. Let $w_{i}^{t}$ be the $i$th row of $W$. Note that $W_{+,x}^{t}W_{+,y}=\sum_{i=1}^{n}1_{w_{i}\cdot x>0}1_{w_{i}\cdot y>0}\cdot w_{i}w_{i}^{t}$, and define

[TABLE]

As $h_{-\epsilon}(z)\geq 1_{z>0}(z)$ and $h_{\epsilon}(z)\leq 1_{z<0}(z)$ for all $z\in\mathbb{R}$, we have that for all nonzero $x,y$ that $G_{\epsilon}(x,y)\preceq W_{+,x}^{t}W_{+,y}\preceq G_{-\epsilon}(x,y)$. Thus, it suffices to establish a matrix upper bound on $G_{-\epsilon}$ and a matrix lower bound on $G_{\epsilon}$.

First we establish a matrix upper bound on $G_{-\epsilon}(x,y)$ uniformly over all nonzero $x,y$. For ease of exposition, in the next two Lemmas, we will take the entries of $W$ to be i.i.d. $\mathcal{N}(0,1)$. In this case, $\mathbb{E}[W_{+,x}^{t}W_{+,y}]=nQ_{x,y}$.

Lemma 12**.**

Fix $0<\epsilon<1$. Let $W\in\mathbb{R}^{n\times k}$ have i.i.d. $\mathcal{N}(0,1)$ entries. If $n>ck\log k$, then with probability at least $1-4ne^{-\gamma k}$,

[TABLE]

Here, $c$ and $\gamma^{-1}$ are constants that depend only polynomially on $\epsilon^{-1}$.

Proof.

Note that the entries of $W$ are assumed to have $\mathcal{N}(0,1)$ entries, and not $\mathcal{N}(0,1/n)$ entries like in most of this paper. In this proof, the values of the constants $c$ and $\gamma$ may change from line to line, but they are all bounded above and below, respectively, by some $\epsilon$-dependent constant. Without loss of generality, let $x,y\in S^{k-1}$.

First, we bound $\mathbb{E}[G_{-\epsilon}(x,y)]$ for fixed $x,y\in S^{k-1}$. Noting that $h_{-\epsilon}(z)\leq 1_{z\geq-\epsilon}(z)=1_{z>0}(z)+1_{-\epsilon\leq z\leq 0}(z)$, we have

[TABLE]

We now bound $\mathbb{E}[1_{-\epsilon\leq w_{i}\cdot x\leq 0}\cdot w_{i}w_{i}^{t}]$. For deriving this bound, we may take $x=e_{1}$ without loss of generality. We have for $\epsilon<1$,

[TABLE]

Thus, $\mathbb{E}[1_{-\epsilon<w_{i}\cdot x<0}\cdot w_{i}w_{i}^{t}]\preceq\frac{\epsilon}{2}I_{k}$ for any $x\neq 0$, resulting in

[TABLE]

Second, we show concentration of $G_{-\epsilon}(x,y)$ for fixed $x,y\in S^{k-1}$. Let $\xi_{i}=\sqrt{h_{-\epsilon}(w_{i}\cdot x)}\sqrt{h_{-\epsilon}(w_{i}\cdot y)}w_{i}$. We have

[TABLE]

Note that $\xi_{i}$ is sub-Gaussian for all $i$ and that the sub-Gaussian norm of $\xi_{i}$ is bounded above by an absolute constant, which we will call $K$. By the first part of Remark 5.40 in [66], there exist constants $c_{K}$ and $\gamma_{K}$ such that for all $t\geq 0$, with probability at least $1-2e^{-\gamma_{K}t^{2}}$,

[TABLE]

If $n>(2/\epsilon)^{2}c_{K}^{2}k$, $t=\epsilon\sqrt{n}/2$, and $\epsilon<1$, we have that with probability at least $1-2e^{-\gamma_{K}\epsilon^{2}n/4}$,

[TABLE]

Third, we bound the Lipschitz constant of $G_{-\epsilon}$. For $\tilde{x},\tilde{y}\in\mathbb{R}^{k}$ we have

[TABLE]

Thus,

[TABLE]

where the second inequality follows because $|h_{-\epsilon}(z)|\leq 1$ for all $z$, and the third inequality follows because $h_{-\epsilon}$ is $1/\epsilon$-Lipschitz.

Let $E_{1}$ be the event that $\|W\|\leq 3\sqrt{n}$. By Corollary 5.35 in [66], we have that $\mathbb{P}(E_{1})\geq 1-2e^{-n/2}$, if $n\geq k$. Let $E_{2}$ be the event that $\max_{i\in[n]}\|w_{i}\|_{2}\leq 2\sqrt{k}$. By a single-tailed variant of Corollary 5.17 in [66], there exists a constant $\gamma_{0}$ such that for fixed $i$, $\|w_{i}\|_{2}\leq 2\sqrt{k}$ with probability at least $1-e^{-\gamma_{0}k}$. Thus, $\mathbb{P}(E_{2})\geq 1-ne^{-\gamma_{0}k}$.

On $E_{1}\cap E_{2}$, we have

[TABLE]

for all $x,y,\tilde{x},\tilde{y}\in S^{k-1}$.

Finally, we complete the proof by a covering argument. Let $\mathcal{N}_{\delta}$ be a $\delta$-net on $S^{k-1}$ such that $|\mathcal{N}_{\delta}|\leq(3/\delta)^{k}$. Take $\delta=\frac{\epsilon^{2}}{36\sqrt{k}}$. Combining (56) and (53), we have

[TABLE]

with probability at least $1-2|\mathcal{N}_{\delta}|e^{-\gamma_{K}\epsilon^{2}n/4}\geq 1-2(\frac{3}{\delta})^{k}e^{-\gamma_{K}\epsilon^{2}n/4}\geq 1-2e^{-\gamma_{K}\epsilon^{2}n/4+k\log(3\cdot 36\sqrt{k}/\epsilon^{2})}$. If $n>\tilde{c}k\log k$, for some $\tilde{c}=\Omega(\epsilon^{-2}\log\epsilon^{-1})$, then this probability is at least $1-2e^{-\tilde{\gamma}n}$ for some $\tilde{\gamma}=O(\epsilon^{2})$. For any $x,y\in S^{k-1}$, let $\tilde{x},\tilde{y}\in\mathcal{N}_{\delta}$ be such that $\|x-\tilde{x}\|_{2}<\delta$ and $\|y-\tilde{y}\|_{2}<\delta$. By (57), we have that

[TABLE]

In conclusion, the result of this lemma holds if $n>(2/\epsilon)^{2}c_{K}^{2}k$ and $n>\tilde{c}k\log k$, with probability at least $1-2e^{-\gamma_{K}\epsilon^{2}n/4}-2e^{-n/2}-ne^{-\gamma_{0}k}-2e^{-\tilde{\gamma}n}>1-4ne^{-\gamma k}$ for some $\gamma=O(\epsilon^{2})$ and $\tilde{c}=\Omega(\epsilon^{-2}\log\epsilon^{-1})$. ∎

Lemma 13**.**

Fix $0<\epsilon<1$. Let $W\in\mathbb{R}^{n\times k}$ have i.i.d. $\mathcal{N}(0,1)$ entries. If $n>ck\log k$, then with probability at least $1-4ne^{-\gamma k}$,

[TABLE]

Here, $c$ and $\gamma^{-1}$ are constants that depend only polynomially on $\epsilon^{-1}$.

Proof.

The proof follows that of Lemma 12 exactly.

First, we bound $\mathbb{E}[G_{\epsilon}(x,y)]$ for fixed $x,y\in S^{k-1}$. For deriving this bound, we take $x=e_{1}$ without loss of generality. Noting that $h_{\epsilon}(z)\geq 1_{z>0}(z)-1_{0\leq z\leq\epsilon}(z)$ for all $z$, we have

[TABLE]

We have

[TABLE]

Thus, $\mathbb{E}[1_{0\leq w_{i}\cdot x\leq\epsilon}\cdot w_{i}w_{i}^{t}]\preceq\frac{\epsilon}{2}I_{k}$ for any $x\neq 0$, resulting in

[TABLE]

Second, the same argument as in Lemma 12 provides that for fixed $x,y\in S^{k-1}$, if $n>(2/\epsilon)^{2}c_{K}^{2}k$, then we have that with probability at least $1-2e^{-\gamma_{K}\epsilon^{2}n/4}$,

[TABLE]

Third, the same argument as in Lemma 12 provides that on the event $E_{1}\cap E_{2}$, we have

[TABLE]

for all $x,y,\tilde{x},\tilde{y}\in S^{k-1}$.

Finally, we complete the proof by an identical covering argument as in Lemma 12. We have that if $n>c_{0}k\log k$ then with probability at least $1-4ne^{-\gamma k}$,

[TABLE]

∎

We may now prove Lemma 11.

Proof of Lemma 11.

It suffices to show

[TABLE]

where $A$ has i.i.d. $\mathcal{N}(0,1)$ entries. The result is immediate if $x=0$ or $y=0$. For all nonzero $x$ and $y$ , we have $G_{\epsilon}(x,y)\preceq W_{+,x}^{t}W_{+,y}\preceq G_{-\epsilon}(x,y)$. The lemma then follows directly from Lemmas 12 and 13. ∎

2.6 Proof of RRIC

We will make use of a standard concentration result used in proving the Restricted Isometry Property from compressed sensing [7].

Lemma 14** (Variant of Lemma 5.1 in [7]).**

Let $A\in\mathbb{R}^{m\times n}$ have i.i.d. $\mathcal{N}(0,1/m)$ entries. Fix $0<\epsilon<1,k<m$. Fix a subspace $T\subset\mathbb{R}^{n}$ of dimension $k$. With probability at least $1-(c_{1}/\epsilon)^{k}e^{-\gamma_{1}\epsilon m}$,

[TABLE]

and

[TABLE]

Let $V=\bigcup_{i=1}^{M}V_{i}$ and $W=\bigcup_{j=1}^{N}W_{j}$, where $V_{i}$ and $W_{j}$ are subspaces of $\mathbb{R}^{n}$ of dimension at most $k$ for all $i,j$. Then,

[TABLE]

with probability at least $1-MN(c_{1}/\epsilon)^{2k}e^{-\gamma_{1}\epsilon m}.$ Here, $c_{1}$ and $\gamma_{1}$ are universal constants.

Proof.

This proof is an immediate extension of Lemma 5.1 in [7]. As $A$ is Gaussian, we may take $T$ to be the span of $k$ standard basis vectors and directly apply the lemma in that paper. The inner product form follows by a standard argument based on the parallelogram identity. The last inequality holds by applying the second inequality to all subspaces of the form $\operatorname{span}(V_{i},W_{j})$, which have dimension at most $2k$, and by applying a union bound. ∎

In order to apply Lemma 14, we now provide an upper bound for the number of subspaces that arise from the objective $f$.

Lemma 15**.**

Let $V$ be a subspace of $\mathbb{R}^{k}$. Let $W\in\mathbb{R}^{n\times k}$ have i.i.d. $\mathcal{N}(0,1/n)$ entries. With probability 1,

[TABLE]

Proof.

Let $\ell=\dim V$. By rotational invariance of Gaussians, we may take $V=\operatorname{span}(e_{1},\ldots,e_{\ell})$ without loss of generality. Without loss of generality, we may let $W$ have dimensions $n\times\ell$ and take $V=\mathbb{R}^{\ell}$.

We will appeal to a classical result from sphere covering [67]. If $n$ hyperplanes in $\mathbb{R}^{\ell}$ contain the origin and are such that the normal vectors to any subset of $\ell$ of those hyperplanes are independent, then the complement of the union of these hyperplanes is partitioned into at most

[TABLE]

disjoint regions. Note that for fixed $W$, $|\{\operatorname{diag}(Wv>0)W\mid v\in\mathbb{R}^{\ell}\}|$ equals the number of binary vectors of the form $(1_{w_{i}\cdot v>0})_{i\in[n]}$ for $v\in S^{\ell-1}$. Each such binary vector corresponds uniquely to one of the disjoint regions given by partitioning the unit sphere in $\mathbb{R}^{\ell}$ by the $n$ half-spaces going through the origin with normal vectors $\{w_{i}\}_{i\in[n]}$. With probability 1, any subset of $\ell$ rows of $W$ are linearly independent, and thus,

[TABLE]

where the first inequality uses the fact that ${n\choose\ell}\leq(en/\ell)^{\ell}$ and the second inequality uses that $2\ell(e/\ell)^{\ell}\leq 10$ for all $\ell\geq 1$.

∎

Lemma 16**.**

Let $W_{i}\in\mathbb{R}^{n_{i}\times n_{i-1}}$ have i.i.d. $\mathcal{N}(0,1/n_{i})$ entries for $i=1,\ldots,d$. Let $k=n_{0}$. Then, with probability 1,

[TABLE]

Proof.

Recall that $W_{1,+,x}=\operatorname{diag}(W_{1}x>0)W_{1}$ and $W_{i,+,x}=\operatorname{diag}(W_{i}W_{i-1,+,x}\cdots W_{1,+,x}x>0)W_{i}$. The case of $i=1$ holds with probability 1 by applying Lemma 15 with $V=\mathbb{R}^{k}$.

Next, we establish the $i=2$ case. Let $\mathcal{W}_{+}=\{W_{+,x}\mid x\neq 0\}$. Note that

[TABLE]

Let the random variable $X_{\hat{W},W_{2}}=|\{\operatorname{diag}(W_{2}v>0)W_{2}\mid v\in\operatorname{range}\hat{W}\}|$. Note that by Lemma 15, for any fixed $\hat{W}$, $\mathbb{P}_{W_{2}}(X_{\hat{W},W_{2}}\leq 10n_{2}^{k})=1$. Hence, conditioned on the probability 1 event $E:=\{|\mathcal{W}_{+}|\leq 10n_{1}^{k}\}$, we have $\mathbb{P}_{W_{2}}(\sum_{\hat{W}\in\mathcal{W}_{+}}X_{\hat{W},W_{2}}\leq 10^{2}n_{1}^{k}n_{2}^{k})=1$. Thus,

[TABLE]

where $\Omega_{1}$ and $\mu_{1}$ are the state space and probability measure for the random variable $W_{1}$. Hence, $|\{W_{2,+,x}\mid x\neq 0\}|\leq 10^{2}n_{1}^{k}n_{2}^{k}$ with probability 1.

The case of larger $i$ follows by repeating the logic above.

∎

We can now show concentration of $v_{x,x_{0}}$ to its expectation with respect to $C$.

Lemma 17**.**

Fix $0<\epsilon<1$. Let $W_{i}\in\mathbb{R}^{n_{i}\times n_{i-1}}$, have i.i.d. $\mathcal{N}(0,1/n_{i})$ entries for $i=1,\ldots,d$. Let $A\in\mathbb{R}^{m\times n_{d}}$ have i.i.d. $\mathcal{N}(0,1/m)$ entries that are independent from all $W_{i}$. If $m>cdk\log(n_{1}n_{2}\cdots n_{d})$, then with probability at least $1-e^{-\gamma m}$,

[TABLE]

Here, $c$ and $\gamma^{-1}$ are constants that depend only polynomially on $\epsilon^{-1}$.

Proof.

For pedagogical purposes, we first establish the lemma in the $d=2$ case. It suffices to show that $\forall x,y,w,v\in S^{k-1}$,

[TABLE]

In order to apply Lemma 14, we will show that $\{W_{2,+,x}W_{1,+,x}w\mid x,w\in S^{k-1}\}$ is a subset of a union of at most $10^{3}(n_{1}^{2}n_{2})^{k}$ subspaces of dimension at most $k$. For fixed $W_{1},W_{2}$, let $\mathcal{A}_{+}=\{W_{1,+,x}\mid x\neq 0\}$ and $\mathcal{B}_{+}=\{W_{2,+,x}\mid x\neq 0\}$. By Lemma 16, there exists a probability 1 event, E, over $(W_{1},W_{2})$ on which $|\mathcal{A}_{+}|\leq 10n_{1}^{k}$ and $|\mathcal{B}_{+}|\leq 10^{2}n_{1}^{k}n_{2}^{k}$. On $E$,

[TABLE]

Note that $\dim\operatorname{range}(W_{2,+,x}W_{1,+,x})\leq k$ for all $x\neq 0$. Hence $\{W_{2,+,x}W_{1,+,x}w\mid x,w\in S^{k-1}\}\subset V$, where $V$ is a union of at most $10^{3}(n_{1}^{2}n_{2})^{k}$ subspaces of dimensionality at most $k$.

By applying the second half of Lemma 14 to the sets $V$ and $V$, we get that for fixed $W_{1},W_{2}$,

[TABLE]

with probability at least $1-10^{3}(c_{1}n_{1}^{2}n_{2}/\epsilon)^{2k}e^{-\gamma_{1}\epsilon m}\geq 1-e^{-\gamma_{2}m}$, provided $m\geq\tilde{c}k\log(n_{1}n_{2})$, for universal constants $c_{1},\gamma_{1}$ and for some $\gamma_{2}=\frac{\gamma_{1}\epsilon}{2},\tilde{c}=\Omega(\epsilon^{-1}\log\epsilon^{-1})$.

Integrating over the probability space of $(W_{1},W_{2})$, independence of $A$ and $(W_{1},W_{2})$ implies that (61) holds for random $(W_{1},W_{2})$ with the same probability bound. Continuing from (61), we have

[TABLE]

$\forall x,y,w,v\in S^{k-1}$ with probability at least $1-e^{-\gamma m}$. for some $\gamma>0$.

For the case of $d\geq 2$, the lemma follows similarly. We have

[TABLE]

on the probability 1 event. The analogous bound to (61) holds with probability at least $1-10^{(d^{2})}(c_{1}n_{1}^{d}n_{2}^{d-1}\cdots n_{d-1}^{2}n_{d}/\epsilon)^{2k}e^{-\gamma_{1}\epsilon m}\geq 1-e^{-\gamma_{2}m}$, provided $m\geq\tilde{c}dk\log(n_{1}n_{2}\cdots n_{d})$, for some \gamma_{2}=\frac{\gamma_{1}\epsilon}{2},\tilde{c}=\Omega(\epsilon^{-1}\log{\color[rgb]{0,0,0}\epsilon^{-1}}).

∎

2.7 Proof of Theorem 6

Theorem 6 can be proved by combining Lemmas 11 and 17.

Acknowledgment

PH is partially supported by NSF Grant DMS-1464525.