Convergence of a Relaxed Variable Splitting Coarse Gradient Descent Method for Learning Sparse Weight Binarized Activation Neural Networks

TL;DR

This paper introduces a new relaxed variable splitting method combining thresholding and coarse gradient descent to efficiently learn sparse, binarized activation CNNs, with proven convergence and explicit error bounds.

Contribution

It develops a novel convergence framework for sparse binarized CNNs using thresholding and coarse gradient descent, with theoretical guarantees and explicit error estimates.

Findings

Proves convergence of the proposed method to a global limit.

Establishes high-probability learnability of sparse binarized CNNs.

Provides explicit error bounds for the learned weights.

Abstract

Sparsification of neural networks is one of the effective complexity reduction methods to improve efficiency and generalizability. Binarized activation offers an additional computational saving for inference. Due to vanishing gradient issue in training networks with binarized activation, coarse gradient (a.k.a. straight through estimator) is adopted in practice. In this paper, we study the problem of coarse gradient descent (CGD) learning of a one hidden layer convolutional neural network (CNN) with binarized activation function and sparse weights. It is known that when the input data is Gaussian distributed, no-overlap one hidden layer CNN with ReLU activation and general weight can be learned by GD in polynomial time at high probability in regression problems with ground truth. We propose a relaxed variable splitting method integrating thresholding and coarse gradient descent. The sparsity in network weight is realized through thresholding during the CGD training process. We prove that under threshholding of $\ell_1, \ell_0,$ and transformed-$\ell_1$ penalties, no-overlap binary activation CNN can be learned with high probability, and the iterative weights converge to a global limit which is a transformation of the true weight under a novel sparsifying operation. We found explicit error estimates of sparse weights from the true weights.

Figures

Tables

Table 1. Table 1: RVSCGD regression: angular errors, loss and sparsity.

measure	s=2	s=4	s=6	s=8	s=10
$\theta (\overline{𝒖},{𝒘}^{\ast })$	0.0076	0.0122	0.0126	0.0133	0.0143
$\theta (\overline{𝒘},{𝒘}^{\ast })$	0.0492	0.0498	0.0499	0.0497	0.0493
$f\left(\overline{𝒖}\right)$	0.0243	0.0388	0.0402	0.0423	0.0456
${\Vert \overline{𝒖}\Vert }_0$	2	4	6	8	10

Equations

$$h(\mathbf{w},\mathbf{Z})=1^T\sigma (\mathbf{Z}\mathbf{w}).$$

$$l(\mathbf{w},\mathbf{Z}):=\frac{1}{m}\sum _{j=1}^m(h(\mathbf{w},{\mathbf{Z}}^j)-h({\mathbf{w}}^{\ast },{\mathbf{Z}}^j){)}^2$$

$$f(\mathbf{w}):={\mathbb{E}}_{\mathbf{Z}\sim \mathcal{G}}\left[(h(\mathbf{w},\mathbf{Z})-h({\mathbf{w}}^{\ast },\mathbf{Z}){)}^2\right]$$

$$\varphi (\mathbf{w})=f(\mathbf{w})+\lambda P(\mathbf{w})$$

$${\mathcal{L}}_{\beta }(\mathbf{u},\mathbf{w})=f(\mathbf{w})+\lambda P(\mathbf{u})+\frac{\beta }{2}\Vert \mathbf{w}-\mathbf{u}{\Vert }^2$$

$$\sigma (x)=\{\begin{array}{ll}0& \text{if }x\le 0\\ 1& \text{if }x>0.\end{array}$$

$$l(\mathbf{w},\mathbf{Z}):=\frac{1}{2}(1^T\sigma (\mathbf{Z}\mathbf{w})-1^T\sigma (\mathbf{Z}{\mathbf{w}}^{\ast }){)}^2$$

$$\underset{\mathbf{w}\in {\mathbb{R}}^d}{\min }\frac{1}{N}\sum _{i=1}^{N}l(\mathbf{w},{\mathbf{Z}}^i).$$

$$\frac{\partial l}{\partial \mathbf{w}}(\mathbf{w},\mathbf{Z})={\mathbf{Z}}^T{\sigma }^{\prime }(\mathbf{Z}\mathbf{w})(\sigma (\mathbf{Z}\mathbf{w})-\sigma (\mathbf{Z}{\mathbf{w}}^{\ast })).$$

$$g(\mathbf{w},\mathbf{Z})=\sqrt{\frac{2}{\pi }}{\mathbf{Z}}^T{\mu }^{\prime }(\mathbf{Z}\mathbf{w})(\sigma (\mathbf{Z}\mathbf{w})-\sigma (\mathbf{Z}{\mathbf{w}}^{\ast }))$$

$$⟨{\mathbb{E}}_{\mathbf{Z}}[g(\mathbf{w},\mathbf{Z})],\nabla f(\mathbf{w})⟩=\frac{\sin (\theta (\mathbf{w},{\mathbf{w}}^{\ast }))}{4{\pi }^2\Vert \mathbf{w}\Vert }\ge 0.$$

$$\eta \Vert {\mathbb{E}}_{\mathbf{Z}}[g({\mathbf{w}}^t,\mathbf{Z})]+\beta ({\mathbf{w}}^t-{\mathbf{u}}^{t+1})\Vert \le \frac{1}{2},\forall t;$$

$$\bar{\mathbf{w}}-\frac{2\pi }{k}\beta (\bar{\mathbf{w}}-S_{\lambda /\beta }(\bar{\mathbf{w}}))=C\bar{\mathbf{w}}$$

$$\Vert {\mathbf{w}}^{\ast }-\bar{\mathbf{w}}\Vert \le \frac{1}{2}\sin \delta \sin \gamma ,$$

$$\Vert {\mathbf{w}}^{\ast }-\bar{\mathbf{u}}\Vert \le \frac{1}{2}\sin \delta \sin \gamma +\frac{\lambda }{\beta }\sqrt{d}.$$

$$\nabla f(\mathbf{w})=\frac{-k}{2\pi \Vert \mathbf{w}\Vert }\frac{\left(\mathbf{I}-\frac{\mathbf{w}{\mathbf{w}}^T}{\Vert \mathbf{w}{\Vert }^2}\right){\mathbf{w}}^{\ast }}{\Vert \left(\mathbf{I}-\frac{\mathbf{w}{\mathbf{w}}^T}{\Vert \mathbf{w}{\Vert }^2}\right){\mathbf{w}}^{\ast }\Vert },$$

$${\mathbb{E}}_{\mathbf{Z}}[g(\mathbf{w},\mathbf{Z})]$$

$$=$$

$$\Vert \nabla f({\mathbf{w}}_1)-\nabla f({\mathbf{w}}_2)\Vert \le L_f\Vert {\mathbf{w}}_1-{\mathbf{w}}_2\Vert$$

$$f({\mathbf{w}}_2)\le f({\mathbf{w}}_1)+⟨\nabla f({\mathbf{w}}_1),{\mathbf{w}}_2-{\mathbf{w}}_1⟩+\frac{L_f}{2}\Vert {\mathbf{w}}_2-{\mathbf{w}}_1{\Vert }^2.$$

$$\Vert {\mathbb{E}}_{\mathbf{Z}}[g({\mathbf{w}}_1,\mathbf{Z})]-{\mathbb{E}}_{\mathbf{Z}}[g({\mathbf{w}}_2,\mathbf{Z})]\Vert \le K\Vert {\mathbf{w}}_1-{\mathbf{w}}_2\Vert$$

$$f({\mathbf{w}}_2)-f({\mathbf{w}}_1)$$

$$\le$$

$${\mathbb{E}}_{\mathbf{Z}}[g({\mathbf{w}}_j,\mathbf{Z})]$$

$$=$$

$$\cos \left(\frac{\theta ({\mathbf{w}}_j,{\mathbf{w}}^{\ast })}{2}\right)=\frac{\frac{1}{2}\Vert {\mathbf{w}}_j+{\mathbf{w}}^{\ast }\Vert }{\Vert {\mathbf{w}}^{\ast }\Vert }=\frac{1}{2}\Vert {\mathbf{w}}_j+{\mathbf{w}}^{\ast }\Vert$$

$${\mathbb{E}}_{\mathbf{Z}}[g({\mathbf{w}}_j,\mathbf{Z})]$$

$$=\frac{k}{2\pi }{\mathbf{w}}_j-\frac{k}{2\pi }{\mathbf{w}}^{\ast }$$

$$\Vert {\mathbb{E}}_{\mathbf{Z}}[g({\mathbf{w}}_1,\mathbf{Z})]-{\mathbb{E}}_{\mathbf{Z}}[g({\mathbf{w}}_2,\mathbf{Z})]\Vert \le K\Vert {\mathbf{w}}_1-{\mathbf{w}}_2\Vert$$

$$f(\mathbf{w})=\frac{1}{8}\left[1^T(I+{11}^T)1-21^T\left(\left(1-\frac{2}{\pi }\theta (\mathbf{w},{\mathbf{w}}^{\ast })\right)I+{11}^T\right)1+1^T(I+{11}^T)1\right]$$

$$f({\mathbf{w}}_2)-f({\mathbf{w}}_1)$$

$$=\frac{1}{4}\left[1^T\left(\left(\frac{2}{\pi }{\theta }_2-\frac{2}{\pi }{\theta }_1\right)I\right)1\right]$$

$$=\frac{k}{2\pi }({\theta }_2-{\theta }_1)$$

Full text

Keywords: Sparse neural networks, sparse penalties, relaxed

variable splitting, thresholding, gradient descent, convergence.

Running Title: Learning sparse neural networks.

AMS Subject Classifications: 90C26, 97R40, 68T05.

1 Introduction

Deep neural networks (DNN) have achieved state-of-the-art performance on many machine learning tasks such as speech recognition [19], computer vision [22], and natural language processing [11]. Training such networks is a problem of minimizing a high-dimensional non-convex and non-smooth objective function, and is often solved by first-order methods such as stochastic gradient descent (SGD). Nevertheless, the success of neural network training remains to be understood from a theoretical perspective. Progress has been made in simplified model problems. [2] showed that even training a 3-node neural network is NP-hard, and [34] showed learning a simple one-layer fully connected neural network is hard for some specific input distributions. Recently, several works ([36]; [5]) focused on the geometric properties of loss functions, which is made possible by assuming that the input data distribution is Gaussian. They showed that SGD with random or zero initialization is able to train a no-overlap neural network in polynomial time.

Another prominent issue is that DNNs contain millions of parameters and lots of redundancies, potentially causing over-fitting and poor generalization [44] besides spending unnecessary computational resources. One way to reduce complexity is to sparsify the network weights using an empirical technique called pruning [23] so that the non-essential ones are zeroed out with minimal loss of performance [17, 38, 26]. Recently a surrogate $\ell_{0}$ regularization approach based on a continuous relaxation of Bernoulli random variables in the distribution sense is introduced with encouraging results on small size image data sets [25]. This motivated our work here to study deterministic regularization of $\ell_{0}$ via its Moreau envelope and related $\ell_{1}$ penalties in a one hidden layer convolutional neural network model [5]. Moreover, we consider binarized activation which further reduces computational costs [42].

The architecture of the network is illustrated in Figure 1 similar to [5]. We consider the convolutional setting in which a sparse filter $\bm{w}\in\mathbb{R}^{d}$ is shared among different hidden nodes. The input sample is $\bm{Z}\in\mathbb{R}^{k\times d}$. Note that this is identical to the one layer non-overlapping case where the input is $x\in\mathbb{R}^{kd}$ with $k$ non-overlapping patches, each of size $d$.

We also assume that the vectors of $\bm{Z}$ are i.i.d. Gaussian random vectors with zero mean and unit variance. Let $\mathcal{G}$ denote this distribution. Finally, let $\sigma$ denote the binarized ReLU activation function, $\sigma(z):=\chi_{\{z>0\}}$ which equals 1 if $z>0$, and 0 otherwise. The output of the network in Figure 1 is given by:

[TABLE]

We address the realizable case, where the response training data is mapped from the input training data $\bm{Z}$ by equation (1) with a ground truth unit weight vector $\bm{w}^{*}$. The input training data is generated by sampling $m$ training points $\bm{Z}^{1},..,\bm{Z}^{m}$ from a Gaussian distribution. The learning problem seeks $\bm{w}$ to minimize the empirical risk function:

[TABLE]

Due to binarized activation, the gradient of $l$ in $w$ is almost everywhere zero, hence in-effective for descent. Instead, an approximate gradient on the coarse scale, the so called coarse gradient (denoted as $\tilde{\nabla}_{\bm{w}}l$) is adopted as proxy and is proved to drive the iterations to global minimum [42].

In the limit $m\uparrow\infty$, the empirical risk $l$ converges to the population risk:

[TABLE]

which is more regular in $\bm{w}$ than $l$. However, the “true gradient” $\nabla_{\bm{w}}f$ is inaccessible in practice. On the other hand, the coarse gradient $\tilde{\nabla}_{\bm{w}}\,l$ in the limit $m\uparrow\infty$ forms an acute angle with the true gradient [42]. Hence the expected coarse gradient descent (CGD) essentially minimizes the population risk $f$ as desired.

Our task is to sparsify $\bm{w}$ in CGD. We note that the iterative thresholding algorithms (IT) are commonly used for retrieving sparse signals ([10, 7, 4, 3, 45] and references therein). In high dimensional setting, IT algorithms provide simplicity and low computational cost, while also promote sparsity of the target vector. We shall investigate the convergence of CGD with simultaneous thresholding for the following objective function

[TABLE]

where $f(\bm{w})$ is the population loss function of the network, and $P$ is $\ell_{0}$, $\ell_{1}$, or the transformed-$\ell_{1}$ (T$\ell_{1}$) function: a one parameter family of bilinear transformations composed with the absolute value function [28, 46]. When acting on vectors, the T$\ell_{1}$ penalty interpolates $\ell_{0}$ and $\ell_{1}$ with thresholding in closed analytical form for any parameter value [45]. The $\ell_{1}$ thresholding function is known as soft-thresholding [10, 13], and that of $\ell_{0}$ the hard-thresholding [4, 3]. The thresholding part should be properly integrated with CGD to be applicable for learning CNNs. As pointed out in [25], it is beneficial to attain sparsity during the optimization (training) process.

Contribution. We propose a Relaxed Variable Splitting (RVS) approach combining thresholding and CGD for minimizing the following augmented objective function

[TABLE]

for a positive parameter $\beta$. We note in passing that minimizing $\mathcal{L}_{\beta}$ in $\bm{u}$ recovers the original objective (4) with penalty $P$ replaced by its Moreau envelope [27]. We shall prove that our algorithm (RVSCGD), alternately minimizing $\bm{u}$ and $\bm{w}$, converges for $\ell_{0}$, $\ell_{1}$, and T$\ell_{1}$ penalties to a global limit $(\bar{\bm{w}},\bar{\bm{u}})$ with high probability. A key estimate is the Lipschitz inequality of the expected coarse gradient (Lemma 5.3). Then the descent of Lagrangian function (5) and the angles between the iterated $\bm{w}$ and $\bm{w}^{*}$ follows. The $\bar{\bm{w}}$ is a novel thresholded version of the true weight $\bm{w}^{*}$ modulo some normalization. The $\bar{\bm{u}}$ is a sparse approximation of $\bm{w}^{*}$. To our best knowledge, this result is the first to establish the convergence of CGD for sparse weight binarized activation networks. In numerical experiments, we observed that the $\bar{\bm{u}}$ limit of RVSCGD with the $\ell_{0}$ penalty recovers sparse $\bm{w}^{*}$ accurately.

Outline. In Section 2, we briefly overview related mathematical results in the study of neural networks and complexity reduction. Preliminaries are in section 3. In Section 4, we state and discuss the main results. The proofs of the main results are in Section 5, and numerical results in Section 6. The conclusion of this paper is in Section 7 and the acknowledgement is in Section 8.

2 Related Work

In recent years, significant progress has been made in the study of convergence in neural network training. From a theoretical point of view, optimizing (training) neural network is a non-convex non-smooth optimization problem. [2, 24, 33] showed that training a neural network is hard in the worst cases. [34] showed that if either the target function or input distribution is “nice”, optimization, algorithms used in practice can succeed. Optimization methods in deep neural networks are often categorized into (stochastic) gradient descent methods and others.

Stochastic gradient descent methods were first proposed by [31]. The popular back-propagation algorithm was introduced in [32]. Since then, many well-known SGD methods with adaptive learning rates were proposed and applied in practice, such as the Polyak momentum [29], AdaGrad [16], RMSProp [37], Adam [21], and AMSGrad [30].

The behavior of gradient descent methods in neural networks is better understood when the input has Gaussian distribution. [36] showed that the population gradient descent can recover the true weight vector with random initialization for one-layer one-neuron model. [5] proved that a convolution filter with non-overlapping input can be learned in polynomial time. [15] showed (stochastic) gradient descent with random initialization can learn the convolutional filter in polynomial time and the convergence rate depends on the smoothness of the input distribution and the closeness of patches. [14] analyzed the polynomial convergence guarantee of randomly initialized gradient descent algorithm for learning a one-hidden-layer convolutional neural network. A hybrid projected SGD (so called BinaryConnect) is widely used for training various weight quantized DNNs [9, 43]. Recently, a Moreau envelope based relaxation method (BinaryRelax) is proposed and analyzed to advance weight quantization in DNN training [41]. Also a blended coarse gradient descent method [42] is introduced to train fully quantized DNNs in weights and activation functions, and overcome vanishing gradients. For earlier work on coarse gradient (a.k.a. straight through estimator), see [18, 20, 6] among others.

Non-SGD methods for deep learning include the Alternating Direction Method of Multipliers (ADMM) to transform a fully-connected neural network into an equality-constrained problem [35]; method of auxiliary coordinates (MAC) to replace a nested neural network with a constrained problem without nesting [8]. [47] handled deep supervised hashing problem by an ADMM algorithm to overcome vanishing gradients.

For a similar model to (5) and treatment in a general context, see [1]; and in image processing, see [40]. For analysis and computation on minimization of (5) to learn a neural network with sparse weight and regular ReLU function, see [12].

3 Preliminaries

Consider a non-overlap one layer convolutional network, where the input feature $\bm{Z}\in\mathbb{R}^{k\times d}$ is i.i.d. Gaussian random vector with zero mean and unit variance. Let $\mathcal{G}$ denote this distribution. Let $\sigma$ be the binarized ReLU function:

[TABLE]

We define the training sample loss by

[TABLE]

where $\bm{w}^{*}\in\mathbb{R}^{d}$ is the underlying (non-zero) teaching parameter. Note that (6) is invariant under scaling $\bm{w}\to\bm{w}/c$, $\bm{w}^{*}\to\bm{w}^{*}/c$, for any scalar $c>0$. Without loss of generality, we assume $\|\bm{w}^{*}\|=1$. Given independent training samples $\{\bm{Z}^{1},...,\bm{Z}^{N}\}$, the associated empirical risk minimization reads

[TABLE]

The empirical risk function in (7) is piece-wise constant and has a.e. zero partial $\bm{w}$ gradient. If $\sigma$ were differentiable, then back-propagation would rely on:

[TABLE]

However, $\sigma$ has zero derivative a.e., rendering (8) inapplicable. We study the coarse gradient descent with $\sigma^{\prime}$ in (8) replaced by the (sub)derivative $\mu^{\prime}$ of the regular ReLU function $\mu(x):=\max(x,0)$. More precisely, we use the following surrogate of $\frac{\partial l}{\partial\bm{w}}(\bm{w},\bm{Z})$:

[TABLE]

with $\mu^{\prime}(x)=\sigma(x)$. The constant $\sqrt{\frac{2}{\pi}}$ will be necessary to give a stronger result for our main findings. To simplify our analysis, we let $N\uparrow\infty$ in (7), so that its coarse gradient approaches $\mathbb{E}_{\bm{Z}}[g(\bm{w},\bm{Z})]$. The following lemma asserts that $\mathbb{E}_{\bm{Z}}[g(\bm{w},\bm{Z})]$ has positive correlation with the true gradient $\nabla f(\bm{w})$, and consequently, $-\mathbb{E}_{\bm{Z}}[g(\bm{w},\bm{Z})]$ gives a reasonable descent direction.

Lemma 3.1**.**

If $\theta(\bm{w},\bm{w}^{*})\in(0,\pi)$, and $\|\bm{w}\|\neq 0$, then the inner product between the expected coarse and true gradient w.r.t. $\bm{w}$ is

[TABLE]

Suppose we want to train the network in a way that $\bm{w}^{t}$ converges to a limit $\bar{\bm{w}}$ in some neighborhood of $\bm{w}^{*}$, and we also want to promote sparsity in the limit $\bar{\bm{w}}$. To this end, a natural function to minimize is (for a parameter $\lambda>0$): $\phi(\bm{w})=f(\bm{w})+\lambda\|\bm{w}\|_{1}$, where other choices of a sparse penalty $P$ include $\ell_{0}$ and $T\ell_{1}$. Our proposed relaxed variable splitting (RVS) proceeds by first extending $\phi$ into a function of two variables $f(\bm{w})+\lambda\|\bm{u}\|_{1}$, then minimizing the Lagrangian function in (5) alternately in $\bm{u}$ and $\bm{w}$. Minimization in $\bm{u}$ is the thresholding operation from penalty $P$, and is in closed form for $\ell_{1}$, $\ell_{0}$ and T$\ell_{1}$. Minimization in $\bm{w}$ is through CGD. The splitting realizes sparsity more effectively than having $P$ under CGD in case of $\ell_{1}$ (T$\ell_{1}$), and bypasses the non-existence of gradient in case of $\ell_{0}$. The resulting RSVCGD Algorithm is as follows:

Here the update of $w^{t}$ has the form $\bm{w}^{t+1}=C^{t}(\bm{w}^{t}-\eta\mathbb{E}_{\bm{Z}}[g(\bm{w}^{t},\bm{Z})]-\eta\beta(\bm{w}^{t}-\bm{u}^{t+1}))$, where $C^{t}$ is some normalization constant. This normalization process is unique to our proposed algorithm, and is distinct from other common descent algorithms, for example ADMM, where the update of $\bm{w}$ has the form $\bm{w}^{t+1}\leftarrow\arg\min_{\bm{w}}\mathcal{L}_{\beta}(\bm{u}^{t+1},\bm{w},\bm{z}^{t})$ and $\bm{z}^{t}$ is the Lagrange multiplier. Since $f$ is non-convex and only Lipschitz differentiable away from zero, convergence analysis of ADMM in this case is beyond the current theory [39]. Here we circumvent the problem by updating $\bm{w}$ via CGD and then normalizing.

Definition 1**.**

The transformed $\ell_{1}$ (T$\ell_{1}$) penalty function on $x\in\mathbb{R}^{d}$ is $P_{a}(x):=\sum_{i=1}^{d}\rho_{a}(x_{i})$, where $\rho_{a}(x)=\frac{(a+1)|x|}{a+|x|}$, for a parameter $a\in(0,+\infty)$.

By varying ’a’, T$\ell_{1}$ interpolates $l_{0}$ and $l_{1}$.

4 Main Results

Theorem 1**.**

*Suppose that the initialization and penalty parameters of the RVSCGD algorithm satisfy:

(i) $\theta(\bm{w}^{0},\bm{w}^{*})\leq\pi-\delta$, for some $\delta>0$,

(ii) $\beta\leq\frac{k\sin\delta}{2\pi}$, and $\lambda<\frac{k}{2\pi\sqrt{d}}$;

and that the learning rate $\eta$ is small so that*

[TABLE]

and $\eta\leq\min\left\{\frac{1}{\beta+L},\frac{2\pi}{k}\right\}$, where $L$ is the Lipschitz constant in Lemma 5.3. Then the Lagrangian $\mathcal{L}_{\beta}(\bm{u}^{t},\bm{w}^{t})$ with $\ell_{1}$ penalty is monotonically decreasing, and $(\bm{u}^{t},\bm{w}^{t})$ converges to a limit point $(\bar{\bm{u}},\bar{\bm{w}})$. Let $\theta:=\theta(\bar{\bm{w}},\bm{w}^{*})$ and $\gamma:=\theta(\bar{\bm{u}},\bar{\bm{w}})$, then $\theta<\delta$, and the critical point $(\bar{\bm{u}},\bar{\bm{w}})$ satisfies

[TABLE]

where $S_{\lambda/\beta}$ is the soft-thresholding operator of $\ell_{1}$, for some positive constant $C$ such that $C\leq\frac{k}{k-2\pi\lambda\sqrt{d}}$; and

[TABLE]

[TABLE]

Remark**.**

The sign of $(\bar{\bm{w}}-S_{\lambda/\beta}(\bar{\bm{w}}))$ agrees with $\bar{\bm{w}}$. Thus $\bar{w}$ is a soft-thresholded version of $\bm{w}^{*}$, after some normalization. The assumption on $\eta$ is reasonable, as will be shown below: $\|\mathbb{E}_{\bm{Z}}[g(\bm{w}^{t},\bm{Z})]\|$ is bounded away from zero, and thus $\|\mathbb{E}_{\bm{Z}}[g(\bm{w}^{t},\bm{Z})]+\beta(\bm{w}^{t}-\bm{u}^{t+1})\|$ is also bounded.

Corollary 1.1**.**

Suppose that the initialization of the RVSCGD algorithm satisfies the conditions in Theorem 1, and that the $\ell_{1}$ penalty is replaced by $\ell_{0}$ or T$\ell_{1}$. Then the RVSCGD iterations converge to a limit point $(\bar{\bm{u}},\bar{\bm{w}})$ satisfying equation (10) with $\ell_{0}$’s hard thresholding operator [3] or T$\ell_{1}$ thresholding [45] replacing $S_{\lambda/\beta}$, and similar bounds (11)-(12) hold.

5 Proof of Main Results

5.1 Proof of Theorem 1

The following Lemmas give the properties of the coarse gradient, as well as an outline for the proof of Theorem 1. The detail of each key step can be found in the proof of the corresponding Lemma.

Lemma 5.1**.**

If every entry of $\bm{Z}$ is i.i.d. sampled from $\mathcal{N}(0,1),\|\bm{w}^{*}\|=1$, and $\|\bm{w}\|\neq 0$, then the true gradient of the population loss $f(\bm{w})$ is

[TABLE]

for $\theta(\bm{w},\bm{w}^{*})\in(0,\pi)$; and the expected coarse gradient w.r.t. $\bm{w}$ is

[TABLE]

Lemma 5.2**.**

*(Properties of true gradient)

Given $\bm{w}_{1},\bm{w}_{2}$ with $\min\{\|\bm{w}_{1}\|,\|\bm{w}_{2}\|\}=c>0$ and $\max\{\|\bm{w}_{1}\|,\|\bm{w}_{2}\|\}=C$, there exists a constant $L_{f}>0$ depends on $c$ and $C$ such that*

[TABLE]

Moreover, we have

[TABLE]

Lemmas 5.1, 5.2 follow directly from [42].

Lemma 5.3**.**

*(Properties of expected coarse gradient)

If $\bm{w}_{1},\bm{w}_{2}$ satisfy $\|\bm{w}_{1}\|,\|\bm{w}_{2}\|=1$, and $\theta(\bm{w}_{1},\bm{w}^{*}),\theta(\bm{w}_{2},\bm{w}^{*})\in(0,\pi)$ with $\theta(\bm{w}_{2},\bm{w}^{*})\leq\theta(\bm{w}_{1},\bm{w}^{*})$, then there exists a constant $K=\frac{k}{2\pi}$ such that*

[TABLE]

Moreover, there exists a constant $L=\frac{k}{4\pi}$ such that

[TABLE]

Remark**.**

Here we show the coarse gradient satisfies the descent condition under very specific conditions. As shown in Lemma 5.4, we have $\theta^{t+1}\leq\theta^{t}$; and the normalization of our RVSCGD model guarantees $\|\bm{w}^{t+1}\|=\|\bm{w}^{t}\|=1$. Thus the result of Lemma 5.3 is sufficient for our proof.

Proof.

First suppose $\|\bm{w}_{1}\|=\|\bm{w}_{2}\|=1$. By Lemma 5.3 of [42], we have

[TABLE]

for $j=1,2$. Consider the plane formed by $\bm{w}_{j}$ and $\bm{w}^{*}$, since $\|\bm{w}^{*}\|=1$, we have an equilateral triangle formed by $\bm{w}_{j}$ and $\bm{w}^{*}$ (See Fig. 2).

Simple geometry shows

[TABLE]

Thus the expected coarse gradient simplifies to

[TABLE]

which implies

[TABLE]

with $K=\frac{k}{2\pi}$. The first claim is proved.

It remains to show the gradient descent inequality. By [42], we have

[TABLE]

Let $\theta_{1}=\theta(\bm{w}_{1},\bm{w}^{*}),\theta_{2}=\theta(\bm{w}_{2},\bm{w}^{*})$. Then

[TABLE]

We will show $f(\bm{w}_{2})-f(\bm{w}_{1})\leq\langle\mathbb{E}_{\bm{Z}}[g(\bm{w}_{1},\bm{Z})],\bm{w}_{2}-\bm{w}_{1}\rangle+L\|\bm{w}_{2}-\bm{w}_{1}\|^{2}$ for $\|\bm{w}_{1}\|=\|\bm{w}_{2}\|=1$ and $\theta_{2}\leq\theta_{1}$. By equation (17),

[TABLE]

It remains to show

[TABLE]

or there exists a constant $K_{1}$ such that

[TABLE]

Notice that by writing $K_{1}=\frac{1}{2}+K_{2}$, we have

[TABLE]

where the last equality follows since $\|\bm{w}_{1}\|=\|\bm{w}_{2}\|=1$ implies $\langle\bm{w}_{1}+\bm{w}_{2},\bm{w}_{2}-\bm{w}_{1}\rangle=0$. On the other hand,

[TABLE]

so it suffices to show there exists a constant $K_{2}$ such that

[TABLE]

Notice the function $\theta\mapsto\theta+\cos\theta$ is monotonically increasing on $[0,\pi]$. For $\theta_{1},\theta_{2}\in[0,\pi]$ with $\theta_{2}\leq\theta_{1}$, the LHS is non-positive, and the inequality holds. Thus one can take $K_{2}=0,K_{1}=\frac{1}{2}$, and $L=\frac{k}{4\pi}$. ∎

Lemma 5.4**.**

*(Angle Descent)

Let $\theta^{t}:=\theta(\bm{w}^{t},\bm{w}^{*})$. If the initialization of the RVSCGD algorithm satisfies*

[TABLE]

then

[TABLE]

Proof.

Due to normalization in the RVSCGD algorithm, $\|\bm{w}^{t}\|=1$ for all $t$. By equation (17), we have

[TABLE]

and the update of $\bm{u}$ is the well-known soft-thresholding of $\bm{w}$ [13, 10]:

[TABLE]

where $S_{\lambda/\beta}(\cdot)$ is the soft-thresholding operator:

[TABLE]

and $S_{\lambda/\beta}(\bm{w})$ applies the thresholding to each component of $\bm{w}$. Then the update of $\bm{w}$ has the form

[TABLE]

for some constant $C^{t}>0$. Suppose the initialization satisfies $\theta(\bm{w}^{0},\bm{w}^{*})\leq\pi-\delta$, for some $\delta>0$. It suffices to show that if $\theta^{t}\leq\pi-\delta$, then $\theta^{t+1}\leq\pi-\delta$. To this end, since $\bm{u}^{t+1}=S_{\lambda/\beta}(\bm{w}^{t})$, we have $\theta(\bm{w}^{t},\bm{u}^{t+1})\leq\frac{\pi}{2}$. Consider the worst case scenario: $\bm{w}^{t},\bm{w}^{*},\bm{u}^{t+1}$ are co-planar with $\theta(\bm{u}^{t+1},\bm{w}^{t})=\frac{\pi}{2}$, and $\bm{w}^{*},\bm{u}^{t+1}$ are on two sides of $\bm{w}^{t}$ (See Fig. 3).

We need $\frac{k}{2\pi}\bm{w}^{*}+\beta\bm{u}^{t+1}$ to be in region I. This condition is satisfied when $\beta$ is small such that

[TABLE]

or

[TABLE]

Since $\|u^{t+1}\|\leq 1$, it suffices to have $\beta\leq\frac{k\sin\delta}{2\pi}$. ∎

Lemma 5.5**.**

*(Lagrangian Descent)

If the initialization of the RVSCGD algorithm satisfies $\eta\leq\frac{1}{\beta+L}$, where $L$ is the Lipschitz constant in Lemma 5.3, then*

[TABLE]

Proof.

By definition of the update on $\bm{u}$, we have $\mathcal{L}_{\beta}(\bm{u}^{t+1},\bm{w}^{t})\leq\mathcal{L}_{\beta}(\bm{u}^{t},\bm{w}^{t})$. It remains to show $\mathcal{L}_{\beta}(\bm{u}^{t+1},\bm{w}^{t+1})\leq\mathcal{L}_{\beta}(\bm{u}^{t+1},\bm{w}^{t})$. First notice that since

[TABLE]

where $C^{t}>0$ is the normalizing constant, thus

[TABLE]

For a fixed $\bm{u}:=\bm{u}^{t+1}$ we have

[TABLE]

Since $\|\bm{w}^{t}\|,\|\bm{w}^{t+1}\|=1$, we know $(\bm{w}^{t+1}-\bm{w}^{t})$ bisects the angle between $\bm{w}^{t+1}$ and $-\bm{w}^{t}$. The assumption $\|\eta\mathbb{E}_{\bm{Z}}[g(\bm{w}^{t},\bm{Z})]+\eta\beta(\bm{w}^{t}-\bm{u}^{t+1})\|\leq\frac{1}{2}$ guarantees $\frac{2}{3}\leq C^{t}\leq 2$ and $\theta(-\bm{w}^{t},\bm{w}^{t+1})<\pi$. It follows that $\theta(\bm{w}^{t+1}-\bm{w}^{t},\bm{w}^{t})$ and $\theta(\bm{w}^{t+1}-\bm{w}^{t},\bm{w}^{t+1})$ are strictly less than $\frac{\pi}{2}$. On the other hand, $\left(\frac{\bm{w}^{t+1}}{C^{t}}-\bm{w}^{t}\right)$ also lies in the plane bounded by $\bm{w}^{t+1}$ and $-\bm{w}^{t}$. Therefore

[TABLE]

This implies $\langle\frac{\bm{w}^{t+1}}{C^{t}}-\bm{w}^{t},\bm{w}^{t+1}-\bm{w}^{t}\rangle\geq 0$. Moreover, when $C^{t}\geq 1$:

[TABLE]

And when $\frac{2}{3}\leq C^{t}\leq 1$:

[TABLE]

Thus we have

[TABLE]

Therefore, if $\eta$ is small so that $\eta\leq\frac{2}{C^{t}(\beta+L)}$ and $\eta\leq\frac{2}{\beta+L}$, the update on $\bm{w}$ will decrease $\mathcal{L}_{\beta}$. Since $C^{t}\leq 2$, the condition is satisfied when $\eta\leq\frac{1}{\beta+L}$. ∎

Lemma 5.6**.**

*(Properties of limit point)

If the initialization of the RVSCGD algorithm satisfies

(i) $\theta(\bm{w}^{0},\bm{w}^{*})\leq\pi-\delta$, for some $\delta>0$

(ii) $\lambda$ is small such that $\frac{2\pi}{k}\lambda\sqrt{d}<1$, $\beta$ is small such that $\beta\leq\frac{k\sin\delta}{2\pi}$

(iii) $\eta$ is small such that $\eta\frac{k}{2\pi}<1$

Let $\theta:=\theta(\bar{\bm{w}},\bm{w}^{*})$ and $\gamma:=\theta(\bar{\bm{u}},\bar{\bm{w}})$, then $(\bm{u}^{t},\bm{w}^{t})$ converges to a critical point $(\bar{\bm{u}},\bar{\bm{w}})$ such that*

[TABLE]

Proof.

Since $\mathcal{L}_{\beta}(\bm{u}^{t},\bm{w}^{t})$ is non-negative, by Lemma 5.4, 5.5, $\mathcal{L}_{\beta}$ converges to some limit $\mathcal{L}$. This implies $(\bm{u}^{t},\bm{w}^{t})$ converges to some stationary point $(\bar{\bm{u}},\bar{\bm{w}})$. By the update of $\bm{w}^{t}$, we have

[TABLE]

for some constant $\bar{C},c_{1},c_{2}>0$, where $c_{2}=\frac{k}{2\pi}$, $c_{1}>0$ due to condition $(iii)$, and $\bar{\bm{u}}=S_{\lambda/\beta}(\bar{\bm{w}})$. For expression (21) to hold, we need

[TABLE]

Expression (22) implies $\bar{\bm{w}},\bar{\bm{u}}$, and $\bm{w}^{*}$ are co-planar. Let $\gamma:=\theta(\bar{\bm{u}},\bar{\bm{w}})$. From expression (22), and the fact that $\|\bar{\bm{w}}\|=\|\bm{w}^{*}\|=1$, we have

[TABLE]

thus

[TABLE]

Recall $\cos(a+b)=\cos a\cos b-\sin a\sin b$. Thus

[TABLE]

By the initialization of $\beta$ and the fact that $\|\bar{\bm{u}}\|<1$, we have

[TABLE]

this implies $\theta<\delta$.

Finally, expression (21) can also be written as

[TABLE]

From expression (24), we see that $\bm{w}^{*}$, after subtracting some vector whose signs agree with $\bar{\bm{w}}$, and whose non-zero components have the same magnitude $\frac{2\pi}{k}\lambda$, is parallel to $\bar{\bm{w}}$. This implies $\bar{\bm{w}}$ is some soft-thresholded version of $\bm{w}^{*}$, modulo normalization. Moreover, since $\left\|\frac{2\pi}{k}\beta(\bar{\bm{w}}-\bar{\bm{u}})\right\|\leq\frac{2\pi}{k}\lambda\sqrt{d}$, for small $\lambda$ such that $\frac{2\pi}{k}\lambda\sqrt{d}<1$, we must have

[TABLE]

On the other hand,

[TABLE]

therefore,

[TABLE]

for some constant $C$ such that $0<C\leq\frac{k}{k-2\pi\lambda\sqrt{d}}$.

Finally, consider the equilateral triangle with sides $\bm{w}^{*},\bar{\bm{w}}$, and $\bm{w}^{*}-\bar{\bm{w}}$. By the law of sines,

[TABLE]

as $\theta$ is small, $\theta(\bar{\bm{w}},\bm{w}^{*}-\bar{\bm{w}})$ is near $\frac{\pi}{2}$. We can assume $\sin\theta(\bar{\bm{w}},\bm{w}^{*}-\bar{\bm{w}})\geq\frac{1}{2}$. Together with expression (23), we have

[TABLE]

The bound on $\|\bm{w}^{*}-\bar{\bm{u}}\|$ follows directly from triangle inequality. ∎

Combining Lemmas 5.1 - 5.6, Theorem 1 is proved.∎

5.2 Proof of Corollary

Lemma 5.7**.**

[45]** Let

[TABLE]

[TABLE]

*where $\phi(x)=\arccos\left(1-\frac{27\lambda a(a+1)}{2(a+|x|)^{3}}\right)$. Then $y_{\lambda}^{*}(x)=\arg\min_{y}f_{\lambda,x}(y)$ is the T$\ell_{1}$ thresholding, equal to $g_{\lambda}(x)$ if $|x|>t$; zero elsewhere. Here $t=\lambda\frac{a+1}{a}$ if $\lambda\leq\frac{a^{2}}{2(a+1)}$; $t=\sqrt{2\lambda(a+1)}-\frac{a}{2}$, elsewhere. *

Lemma 5.8**.**

[3*]** Let $f_{\lambda,x}(y)=\frac{1}{2}(y-x)^{2}+\lambda\,\|y\|_{0}.$ Then $y_{\lambda}^{*}(x)=\arg\min_{y}f_{\lambda,x}(y)$ is the $\ell_{0}$ hard thresholding $y^{*}_{\lambda}(x)=x$, if $|x|>\sqrt{2\lambda}$; zero elsewhere. *

We proceed by an outline similar to the proof of Theorem 1:

Step 1. First we show that $L_{\beta,T\ell_{1}}(\bm{u}^{t},\bm{w}^{t})$ and $L_{\beta,0}(\bm{u}^{t},\bm{w}^{t})$ both decrease under the update of $\bm{u}^{t}$ and $\bm{w}^{t}$. To see this, notice that the update on $\bm{u}^{t}$ decreases $L_{\beta,T\ell_{1}}(\bm{u}^{t},\bm{w}^{t})$ and $L_{\beta,0}(\bm{u}^{t},\bm{w}^{t})$ by definition. Then, for a fixed $\bm{u}=\bm{u}^{t+1}$, the update on $\bm{w}^{t}$ decreases $L_{\beta,T\ell_{1}}(\bm{u}^{t},\bm{w}^{t})$ and $L_{\beta,0}(\bm{u}^{t},\bm{w}^{t})$ by a similar argument to that found in Theorem 1.

Step 2. Next, we show $\theta(\bm{w}^{t},\bm{w}^{*})\leq\pi-\delta$, for some $\delta>0$, for all $t$, with initialization $\theta(\bm{w}^{0},\bm{w}^{*})=\pi-\delta$. For $L_{\beta,T\ell_{1}}(\bm{u}^{t},\bm{w}^{t})$, by Lemma 5.7, we have

[TABLE]

And for $L_{\beta,0}(\bm{u}^{t},\bm{w}^{t})$ , by Lemma 5.8,

[TABLE]

In both cases, each component of $\bm{u}^{t+1}$ is a thresholded version of the corresponding component of $\bm{w}^{t}$. This implies $\theta(\bm{u}^{t+1},\bm{w}^{t})\leq\frac{\pi}{2}$, and thus the argument in Theorem 1 follows through, and we have $\theta(\bm{w}^{t},\bm{w}^{*})\leq\pi-\theta$, for all $t$.

Step 3. Finally, the equilibrium condition from equation $\eqref{ParallelCondition}$ still holds for the critical point, and a similar argument shows that $\theta(\bar{\bm{w}},\bm{w}^{*})<\delta$. ∎

6 Numerical Experiment

Let $(k,d)=(20,50)$, $\beta=10^{-3}$, $\eta=10^{-5}$, $\|\bm{w}^{*}\|_{0}=s\ll d$, $\bm{w}^{*}=s^{-1/2}(1,\cdots,1,0,\cdots,0)$. Table 1 shows result by RVSCGD with $\ell_{0}$ penalty, $\lambda=10^{-4}$, random start in $\bm{w}$. The sparsity $s$ is exactly recovered in $\bar{\bm{u}}$, with small angular errors and loss values. The angle descent vs. iterations is seen in Fig. 4.

7 Conclusion

We introduced a variable splitting coarse gradient descent method to learn a one-hidden layer neural network with sparse weight and binarized activation in a regression setting. The proof is based on the descent of a Lagrangian function and the angle between the sparse and true weights, and applies to $\ell_{1}$, $\ell_{0}$ and T$\ell_{1}$ sparse penalties. We plan to extend our work to a classification setting in the future.

8 Acknowledgement

The work was partially supported by NSF grant IIS-1632935. The authors thank Dr. Penghang Yin for the helpful comments.