Shakeout: A New Approach to Regularized Deep Neural Network Training

TL;DR

Shakeout is a novel regularization method for deep neural networks that enhances or reverses unit contributions, combining multiple regularizers and outperforming Dropout in reducing overfitting, inducing sparsity, and stabilizing training.

Contribution

This paper introduces Shakeout, a new regularization technique that improves upon Dropout by adaptively combining L0, L1, and L2 regularization effects in deep neural network training.

Findings

Shakeout outperforms Dropout on image classification tasks.

Shakeout induces sparser weights and the grouping effect.

Shakeout reduces training instability.

Abstract

Recent years have witnessed the success of deep neural networks in dealing with a plenty of practical problems. Dropout has played an essential role in many successful deep neural networks, by inducing regularization in the model training. In this paper, we present a new regularized training approach: Shakeout. Instead of randomly discarding units as Dropout does at the training stage, Shakeout randomly chooses to enhance or reverse each unit's contribution to the next layer. This minor modification of Dropout has the statistical trait: the regularizer induced by Shakeout adaptively combines $L_0$, $L_1$ and $L_2$ regularization terms. Our classification experiments with representative deep architectures on image datasets MNIST, CIFAR-10 and ImageNet show that Shakeout deals with over-fitting effectively and outperforms Dropout. We empirically demonstrate that Shakeout leads to sparser weights under both unsupervised and supervised settings. Shakeout also leads to the grouping effect of the input units in a layer. Considering the weights in reflecting the importance of connections, Shakeout is superior to Dropout, which is valuable for the deep model compression. Moreover, we demonstrate that Shakeout can effectively reduce the instability of the training process of the deep architecture.

Figures

Tables

Table 1. TABLE I: The architecture of convolutional neural network adopted for MNIST classification experiment

Layer	1	2	3	4
Type	conv.	conv.	FC	FC
Channels	20	50	500	10
Filter size	$5\times 5$	$5\times 5$	-	-
Conv. stride	1	1	-	-
Pooling type	max	max	-	-
Pooling size	$2\times 2$	$2\times 2$	-	-
Pooling stride	2	2	-	-
Non-linear	ReLU	ReLU	ReLU	Softmax

Table 2. TABLE II: Classification on MNIST using training sets of different sizes: fully-connected neural network

Size	std-BP	Dropout	Shakeout
500	13.66$\pm$0.66	11.76$\pm$0.09	10.81$\pm$0.32
1000	8.49$\pm$0.23	8.05$\pm$0.05	7.19$\pm$0.15
3000	5.54$\pm$0.09	4.87$\pm$0.06	4.60$\pm$0.07
8000	3.57$\pm$0.14	2.95$\pm$0.05	2.96$\pm$0.09
20000	2.28$\pm$0.09	1.82$\pm$0.07	1.92$\pm$0.06
50000	1.55$\pm$0.03	1.36$\pm$0.03	1.35$\pm$0.07

Table 3. TABLE III: Classification on MNIST using training sets of different sizes: convolutional neural network

Size	std-BP	Dropout	Shakeout
500	9.76$\pm$0.26	6.16$\pm$0.23	4.83$\pm$0.11
1000	6.73$\pm$0.12	4.01$\pm$0.16	3.43$\pm$0.06
3000	2.93$\pm$0.10	2.06$\pm$0.06	1.86$\pm$0.13
8000	1.70$\pm$0.03	1.23$\pm$0.13	1.31$\pm$0.06
20000	0.97$\pm$0.01	0.83$\pm$0.06	0.77$\pm$$0$.001
50000	0.78$\pm$0.05	0.62$\pm$0.04	0.58$\pm$0.10

Table 4. TABLE IV: Classification on CIFAR-10 using training sets of different sizes: AlexFastNet

Size	std-BP	Dropout	Shakeout
300	68.26$\pm$0.57	65.34$\pm$0.75	63.71$\pm$0.28
700	59.78$\pm$0.24	56.04$\pm$0.22	54.66$\pm$0.22
2000	50.73$\pm$0.29	46.24$\pm$0.49	44.39$\pm$0.41
5500	41.41$\pm$0.52	36.01$\pm$0.13	34.54$\pm$0.31
15000	32.53$\pm$0.25	27.28$\pm$0.26	26.53$\pm$0.17
40000	24.48$\pm$0.23	20.50$\pm$0.32	20.56$\pm$0.12

Table 5. TABLE V: Classification on CIFAR-10 using training sets of different sizes: WRN-16-4

Size	std-BP	Dropout	Shakeout
15000	20.95	15.05	14.68
40000	15.37	9.32	9.01
50000	14.39	8.03	7.97

Equations

$$\theta$$

$$\{\begin{array}{ll}{\tilde{w}}_j\leftarrow -c{s}_j,& \text{if }{r}_j=0\text{(A)}\\ {\tilde{w}}_j\leftarrow (w_j+c\tau s_j)/(1-\tau )& \text{otherwise}\text{(B)}\end{array}$$

$$P(y|\mathbf{x},\mathbf{w})$$

$$\theta$$

$$l(\mathbf{w},\mathbf{x},y)$$

$$A(\theta )$$

$$l_{\text{sko}}(\mathbf{w},\mathbf{x},y,\mathbf{r}):=l(\mathbf{w},\tilde{\mathbf{x}},y)$$

$${\mathbb{E}}_{\mathbf{r}}[l_{\text{sko}}(\mathbf{w},\mathbf{x},y,\mathbf{r})]=l(\mathbf{w},\mathbf{x},y)+\pi (\mathbf{w})$$

$$\pi (\mathbf{w})$$

$$\pi (\mathbf{w})=\tau \sum _{j=1}^{p}A({\theta }_{j-})+(1-\tau )\sum _{j=1}^{p}A({\theta }_{j+})-pA(\theta )$$

$$\mathbb{E}[(\tilde{\theta }-\theta {)}^k]$$

$$\pi (\mathbf{w})$$

$$\pi (\mathbf{w})=\tau \sum _{j=1}^{p}A({\theta }_{j-})+(1-\tau )\sum _{j=1}^{p}A({\theta }_{j+})-pA(\theta )$$

$$\pi (\mathbf{w})=\frac{\tau }{2(1-\tau )}{\Vert \mathbf{x}\circ (\mathbf{w}+c\mathbf{s})\Vert }_2^2$$

$$\sum _{j=1}^p{x}_j^2{w}_j^2+2c\sum _{j=1}^p{x}_j^2|w_j|+c^2\sum _{j=1}^p{x}_j^2{1}_{w_j\ne 0}[w_j]$$

$$\pi (\mathbf{w})=\sum _{j=1}^p\ln (\frac{(1+\exp ({\theta }_{j-}){)}^{\tau }(1+\exp ({\theta }_{j+}){)}^{1-\tau }}{1+\exp (\theta )})$$

$${\pi }_{approx}(\mathbf{w})=\frac{\tau }{2(1-\tau )}{A}^{{}^{\prime \prime }}(\theta ){\Vert \mathbf{x}\circ (\mathbf{w}+c\mathbf{s})\Vert }_2^2$$

$$u_i$$

$$x_i^{{}^{\prime }}$$

$$\frac{\partial u_i}{\partial x_j}$$

$$\frac{\partial u_i}{\partial W_{ij}}$$

$${\mathbf{U}}_i=\sum _j({\mathbf{X}}_j\circ {\mathbf{R}}_j)\ast {\mathbf{W}}_{ij}+c({\mathbf{X}}_j\circ ({\mathbf{R}}_j-1))\ast {\mathbf{S}}_{ij}+b_i$$

$${\mathbf{X}}_i^{{}^{\prime }}=f({\mathbf{U}}_i)$$

$$\frac{\partial {\mathbf{U}}_i(a,b)}{\partial {\mathbf{X}}_j(a-a^{{}^{\prime }},b-b^{{}^{\prime }})}$$

$$c{\mathbf{S}}_{ij}(a^{{}^{\prime }},b^{{}^{\prime }}))-c{\mathbf{S}}_{ij}(a^{{}^{\prime }},b^{{}^{\prime }})$$

$$\begin{gathered} \frac{\partial {\mathbf{U}}_i(a,b)}{\partial {\mathbf{W}}_{ij}(a^{{}^{\prime }},b^{{}^{\prime }})}={\mathbf{X}}_j(a-a^{{}^{\prime }},b-b^{{}^{\prime }})({\mathbf{R}}_j(a-a^{{}^{\prime }},b-b^{{}^{\prime }}) \\ +c({\mathbf{R}}_j(a-a^{{}^{\prime }},b-b^{{}^{\prime }})-1)\frac{d{\mathbf{S}}_{ij}(a^{{}^{\prime }},b^{{}^{\prime }})}{d{\mathbf{W}}_{ij}(a^{{}^{\prime }},b^{{}^{\prime }})}) \end{gathered}$$

$$R.A.L(m^{{}^{\prime }})=\frac{Accu.(m=0)-Accu.(m^{{}^{\prime }})}{Accu.(m=0)}$$

$$\underset{G}{\min }\underset{D}{\max }V(D,G)=\mathbb{E}[\log [D(\mathbf{x})]+\log [1-D(G(\mathbf{z}))]]$$

Full text

1 Introduction

Deep neural networks have recently achieved impressive success in a number of machine learning and pattern recognition tasks and been under intensive research [1, 2, 3, 4, 5, 6, 7]. Hierarchical neural networks have been known for decades, and there are a number of essential factors contributing to its recent rising, such as the availability of big data and powerful computational resources. However, arguably the most important contributor to the success of deep neural network is the discovery of efficient training approaches [8, 9, 10, 11, 12].

A particular interesting advance in the training techniques is the invention of Dropout [13]. At the operational level, Dropout adjusts the network evaluation step (feed-forward) at the training stage, where a portion of units are randomly discarded. The effect of this simple trick is impressive. Dropout enhances the generalization performance of neural networks considerably, and is behind many record-holders of widely recognized benchmarks [14, 2, 15]. The success has attracted much research attention, and found applications in a wider range of problems [16, 17, 18]. Theoretical research from the viewpoint of statistical learning has pointed out the connections between Dropout and model regularization, which is the de facto recipe of reducing over-fitting for complex models in practical machine learning. For example, Wager et al. [16] showed that for a generalized linear model (GLM), Dropout implicitly imposes an adaptive $L_{2}$ regularizer of the network weights through an estimation of the inverse diagonal Fisher information matrix.

Sparsity is of vital importance in deep learning. It is straightforward that through removing unimportant weights, deep neural networks perform prediction faster. Additionally, it is expected to obtain better generalization performance and reduce the number of examples needed in the training stage [19]. Recently much evidence has shown that the accuracy of a trained deep neural network will not be severely affected by removing a majority of connections and many researchers focus on the deep model compression task [20, 21, 22, 23, 24, 25]. One effective way of compression is to train a neural network, prune the connections and fine-tune the weights iteratively [21, 22]. However, if we can cut the connections naturally via imposing sparsity-inducing penalties in the training process of a deep neural network, the work-flow will be greatly simplified.

In this paper, we propose a new regularized deep neural network training approach: Shakeout, which is easy to implement: randomly choosing to enhance or reverse each unit’s contribution to the next layer in the training stage. Note that Dropout can be considered as a special “flat” case of our approach: randomly keeping (enhance factor is $1$) or discarding (reverse factor is [math]) each unit’s contribution to the next layer. Shakeout enriches the regularization effect. In theory, we prove that it adaptively combines $L_{0}$, $L_{1}$ and $L_{2}$ regularization terms. $L_{0}$ and $L_{1}$ regularization terms are known as sparsity-inducing penalties. The combination of sparsity-inducing penalty and $L_{2}$ penalty of the model parameters has shown to be effective in statistical learning: the Elastic Net [26] has the desirable properties of producing sparse models while maintaining the grouping effect of the weights of the model. Because of the randomly “shaking” process and the regularization characteristic pushing network weights to zero, our new approach is named “Shakeout”.

As discussed above, it is expected to obtain much sparser weights using Shakeout than using Dropout because of the combination of $L_{0}$ and $L_{1}$ regularization terms induced in the training stage. We apply Shakeout on one-hidden-layer autoencoder and obtain much sparser weights than that resulted by Dropout. To show the regularization effect on the classification tasks, we conduct the experiments on image datasets including MNIST, CIFAR-10 and ImageNet with the representative deep neural network architectures. In our experiments we find that by using Shakeout, the trained deep neural networks always outperform those by using Dropout, especially when the data is scarce. Besides the fact that Shakeout leads to much sparser weights, we also empirically find that it groups the input units of a layer. Due to the induced $L_{0}$ and $L_{1}$ regularization terms, Shakeout can result in the weights reflecting the importance of the connections between units, which is meaningful for conducting compression. Moreover, we demonstrate that Shakeout can effectively reduce the instability of the training process of the deep architecture.

This journal paper extends our previous work [27] theoretically and experimentally. The main extensions are listed as follows:

1. we derive the analytical formula for the regularizer induced by Shakeout in the context of GLM and prove several important properties;
2. we conduct experiments using Wide Residual Network [15] on CIFAR-10 to show Shakeout outperforms Dropout and standard back-propagation in promoting the generalization performance of a much deeper architecture;
3. we conduct experiments using AlexNet [14] on ImageNet dataset with Shakeout and Dropout. Shakeout obtains comparable classification performance to Dropout, but with superior regularization effect;
4. we illustrate that Shakeout can effectively reduce the instability of the training process of the deep architecture. Moreover, we provide a much clearer and detailed description of Shakeout, derive the forward-backward update rule for deep convolutional neural networks with Shakeout, and give several recommendations to help the practitioners make full use of Shakeout.

In the rest of the paper, we give a review about the related work in Section 2. Section 3 presents Shakeout in detail, along with theoretical analysis of the regularization effect induced by Shakeout. In Section 4, we first demonstrate the regularization effect of Shakeout on the autoencoder model. The classification experiments on MNIST , CIFAR-10 and ImageNet illustrate that Shakeout outperforms Dropout considering the generalization performance, the regularization effect on the weights, and the stabilization effect on the training process of the deep architecture. Finally, we give some recommendations for the practitioners to make full use of Shakeout.

2 Related Work

Deep neural networks have shown their success in a wide variety of applications. One of the key factors contributes to this success is the creation of powerful training techniques. The representative power of the network becomes stronger as the architecture gets deeper [9]. However, millions of parameters make deep neural networks easily over-fit. Regularization [28, 16] is an effective way to obtain a model that generalizes well. There exist many approaches to regularize the training of deep neural networks, like weight decay [29], early stopping [30], etc. Shakeout belongs to the family of regularized training techniques.

Among these regularization techniques, our work is closely related to Dropout [13]. Many subsequent works were devised to improve the performance of Dropout [31, 32, 33]. The underlying reason why Dropout improves performance has also attracted the interest of many researchers. Evidence has shown that Dropout may work because of its good approximation to model averaging and regularization on the network weights [34, 35, 36]. Srivastava [34] and Warde-Farley [35] exhibited through experiments that the weight scaling approximation is an accurate alternative for the geometric mean over all possible sub-networks. Gal et al. [37] claimed that training the deep neural network with Dropout is equivalent to performing variational inference in a deep Gaussian Process. Dropout can also be regarded as a way of adding noise into the neural network. By marginalizing the noise, Srivastava [34] proved for linear regression that the deterministic version of Dropout is equivalent to adding an adaptive $L_{2}$ regularization on the weights. Furthermore, Wager [16] extended the conclusion to generalized linear models (GLMs) using a quadratic approximation to the induced regularizer. The inductive bias of Dropout was studied by Helmbold et al. [38] to illustrate the properties of the regularizer induced by Dropout further. In terms of implicitly inducing regularizer of the network weights, Shakeout can be viewed as a generalization of Dropout. It enriches the regularization effect of Dropout, i.e. besides the $L_{2}$ regularization term, it also induces the $L_{0}$ and $L_{1}$ regularization terms, which may lead to sparse weights of the model.

Due to the implicitly induced $L_{0}$ and $L_{1}$ regularization terms, Shakeout is also related to sparsity-inducing approaches. Olshausen et al. [39] introduced the concept of sparsity in computational neuroscience and proposed the sparse coding method in the visual system. In machine learning, the sparsity constraint enables a model to capture the implicit statistical data structure, performs feature selection and regularization, compresses the data at a low loss of the accuracy, and helps us to better understand our models and explain the obtained results. Sparsity is one of the key factors underlying many successful deep neural network architectures [40, 41, 42, 2] and training algorithms [43][44]. A Convolutional neural network is much sparser than the fully-connected one, which results from the concept of local receptive field [40]. Sparsity has been a design principle and motivation for Inception-series models [41, 42, 2]. Besides working as the heuristic principle of designing a deep architecture, sparsity often works as a penalty induced to regularize the training process of a deep neural network. There exist two kinds of sparsity penalties in deep neural networks, which lead to the activity sparsity [43][44] and the connectivity sparsity [45] respectively. The difference between Shakeout and these sparsity-inducing approaches is that for Shakeout, the sparsity is induced through simple stochastic operations rather than manually designed architectures or explicit norm-based penalties. This implicit way enables Shakeout to be easily optimized by stochastic gradient descent (SGD) ${-}$ the representative approach for the optimization of a deep neural network.

3 Shakeout

Shakeout applies on the weights in a linear module. The linear module, i.e. weighted sum,

[TABLE]

is arguably the most widely adopted component in data models. For example, the variables $x_{1}$, $x_{2}$, $\dots$, $x_{p}$ can be input attributes of a model, e.g. the extracted features for a GLM, or the intermediate outputs of earlier processing steps, e.g. the activations of the hidden units in a multilayer artificial neural network. Shakeout randomly modifies the computation in Eq. (1). Specifically, Shakeout can be realized by randomly modifying the weights

Step 1: Draw $r_{j}$, where $\begin{cases}P(r_{j}=0)&=\tau\\ P(r_{j}=\frac{1}{1-\tau})&=1-\tau\end{cases}$ .

Step 2: Adjust the weight according to $r_{j}$,

[TABLE]

where $s_{j}=\textrm{sgn}(w_{j})$ takes $\pm 1$ depending on the sign of $w_{j}$ or takes 0 if $w_{j}=0$. As shown above, Shakeout chooses (randomly by drawing $r$) between two fundamentally different ways to modify the weights. Modification (A) is to set the weights to constant magnitudes, despite their original values except for the signs (to be opposite to the original ones). Modification (B) updates the weights by a factor $(1-\tau)^{-1}$ and a bias depending on the signs. Note both (A) and (B) preserve zero values of the weights, i.e. if $w_{j}=0$ then $\tilde{w}_{j}=0$ with probability 1. Let $\tilde{\theta}=\tilde{\boldsymbol{w}}^{T}\boldsymbol{x}$, and Shakeout leaves $\theta$ unbiased, i.e. $\mathbb{E}[\tilde{\theta}]=\theta$. The hyper-parameters $\tau\in(0,1)$ and $c\in(0,+\infty)$ configure the property of Shakeout.

Shakeout is naturally connected to the widely adopted operation of Dropout [13, 34]. We will show that Shakeout has regularization effect on model training similar to but beyond what is induced by Dropout. From an operational point of view, Fig. 1 compares Shakeout and Dropout. Note that Shakeout includes Dropout as a special case when the hyper-parameter $c$ in Shakeout is set to zero.

When applied at the training stage, Shakeout alters the objective $-$ the quantity to be minimized $-$ by adjusting the weights. In particular, we will show that Shakeout (with expectation over the random switch) induces a regularization term effectively penalizing the magnitudes of the weights and leading to sparse weights. Shakeout is an approach designed for helping model training, when the models are trained and deployed, one should relieve the disturbance to allow the model work with its full capacity, i.e. we adopt the resulting network without any modification of the weights at the test stage.

3.1 Regularization Effect of Shakeout

Shakeout randomly modifies the weights in a linear module, and thus can be regarded as injecting noise into each variable $x_{j}$, i.e. $x_{j}$ is randomly scaled by $\gamma_{j}$: $\tilde{x}_{j}=\gamma_{j}x_{j}$. Note that $\gamma_{j}=r_{j}+\frac{c(r_{j}-1)}{|w_{j}|}$, the modification of $x_{j}$ is actually determined by the random switch ${r_{j}}$. Shakeout randomly chooses to enhance (i.e. when $r_{j}=\frac{1}{1-\tau}$, $\gamma_{j}>\frac{1}{1-\tau}$) or reverse (i.e. when $r_{j}=0$, $\gamma_{j}<0$) each original variable $x_{j}$’s contribution to the output at the training stage (see Fig. 1). However, the expectation of $\tilde{x}_{j}$ over the noise remains unbiased, i.e. $\mathbb{E}_{r_{j}}[\tilde{x}_{j}]=x_{j}$.

It is well-known that injecting artificial noise into the input features will regularize the training objective [16, 46, 47], i.e. $\mathbb{E}_{\boldsymbol{r}}[\ell(\boldsymbol{w},\tilde{\boldsymbol{x}},y)]=\ell(\boldsymbol{w},\boldsymbol{x},y)+\pi(\boldsymbol{w})$, where $\tilde{\boldsymbol{x}}$ is the feature vector randomly modified by the noise induced by $\boldsymbol{r}$. The regularization term $\pi(\boldsymbol{w})$ is determined by the characteristic of the noise. For example, Wager et al.[16] showed that Dropout, corresponding to inducing blackout noise to the features, helps introduce an adaptive $L_{2}$ penalty on $\boldsymbol{w}$. In this section we illustrate how Shakeout helps regularize model parameters $\boldsymbol{w}$ using an example of GLMs.

Formally, a GLM is a probabilistic model of predicting target $y$ given features $\boldsymbol{x}=[x_{1},\dots,x_{p}]$, in terms of the weighted sum in Eq. (1):

[TABLE]

With different $h(\cdot)$ and $g(\cdot)$ functions, GLM can be specialized to various useful models or modules, such as logistic regression model or a layer in a feed-forward neural network. However, roughly speaking, the essence of a GLM is similar to that of a standard linear model which aims to find weights $w_{1},\dots,w_{p}$ so that $\theta=\boldsymbol{w}^{T}\boldsymbol{x}$ aligns with $y$ (functions $h(\cdot)$ and $g(\cdot)$ are independent of $\boldsymbol{w}$ and $y$ respectively). The loss function of a GLM with respect to $\boldsymbol{w}$ is defined as

[TABLE]

The loss (3) is the negative logarithm of probability (2), where we keep only terms relevant to $\boldsymbol{w}$.

Let the loss with Shakeout be

[TABLE]

where $\boldsymbol{r}=[r_{1},\dots,r_{p}]^{T}$, and $\tilde{\boldsymbol{x}}=[\tilde{x}_{1},\dots,\tilde{x}_{p}]^{T}$ represents the features randomly modified with $\boldsymbol{r}$.

Taking expectation over $\boldsymbol{r}$, the loss with Shakeout becomes

[TABLE]

where

[TABLE]

is named Shakeout regularizer. Note that if $A(\theta)$ is $k$-th order derivable, let the $k^{{}^{\prime}}$ order derivative $A^{(k^{{}^{\prime}})}(\theta)=0$ where $k^{{}^{\prime}}>k$, to make the denotation simple.

Theorem 1

Let $q_{j}=x_{j}(w_{j}+cs_{j})$, $\theta_{j-}=\theta-q_{j}$ and $\theta_{j+}=\theta+\frac{\tau}{1-\tau}q_{j}$, then Shakeout regularizer $\mathrm{\pi(\boldsymbol{w})}$ is

[TABLE]

Proof:

Note that $\tilde{\theta}-\theta=\sum_{j=1}^{p}q_{j}(r_{j}-1)$, then for Eq. (6)

[TABLE]

Because arbitrary two random variables $r_{j_{m_{1}}}$, $r_{j_{m_{2}}}$ are independent unless $j_{m_{1}}=j_{m_{2}}$ and $\forall r_{j_{m}}$, $\mathbb{E}[r_{j_{m}}-1]=0$, then

[TABLE]

Then

[TABLE]

Further, let $\theta_{j-}=\theta-q_{j}$, $\theta_{j+}=\theta+\frac{\tau}{1-\tau}q_{j}$, $\mathrm{\pi(\boldsymbol{w})}$ becomes

[TABLE]

The theorem is proved. ∎

We illustrate several properties of Shakeout regularizer based on Eq. (7). The proof of the following propositions can be found in the appendices.

Proposition 1

$\pi(\boldsymbol{0})=0$**

Proposition 2

If $A(\theta)$ is convex, $\mathrm{\pi(\boldsymbol{w})}\geq 0$.

Proposition 3

Suppose $\exists j$, $x_{j}w_{j}\neq 0$. If $A(\theta)$ is convex, $\mathrm{\pi(\boldsymbol{w})}$ monotonically increases with $\tau$. If $A^{{}^{\prime\prime}}(\theta)>0$, $\mathrm{\pi(\boldsymbol{w})}$ monotonically increases with $c$.

Proposition 3 implies that the hyper-parameters $\tau$ and $c$ relate to the strength of the regularization effect. It is reasonable because higher $\tau$ or $c$ means the noise injected into the features $\boldsymbol{x}$ has larger variance.

Proposition 4

Suppose i) $\forall j\neq j^{{}^{\prime}}$, $x_{j}w_{j}=0$, and ii) $x_{j^{{}^{\prime}}}\neq 0$.

Then

i) if $A^{{}^{\prime\prime}}(\theta)>0$, $\begin{cases}\frac{\partial\mathrm{\pi(\boldsymbol{w})}}{\partial w_{j^{\prime}}}>0,&\textrm{when}\ w_{j^{\prime}}>0\\ \frac{\partial\mathrm{\pi(\boldsymbol{w})}}{\partial w_{j^{\prime}}}<0,&\textrm{when}\ w_{j^{\prime}}<0\end{cases}$

ii) if $\lim_{|\theta|\rightarrow\infty}A^{{}^{\prime\prime}}(\theta)=0$, $\lim_{|w_{j^{\prime}}|\rightarrow\infty}\frac{\partial\mathrm{\pi(\boldsymbol{w})}}{\partial w_{j^{\prime}}}=0$

Proposition 4 implies that under certain conditions, starting from a zero weight vector, Shakeout regularizer penalizes the magnitude of $w_{j^{{}^{\prime}}}$ and its regularization effect is bounded by a constant value. For example, for logistic regression, $\mathrm{\pi(\boldsymbol{w})}\leq\tau\ln(1+\exp(c|x_{j^{{}^{\prime}}}|))$, which is illustrated in Fig. 2. This bounded property has been proved to be useful: capped-norm [48] is more robust to outliers than the traditional $L_{1}$ or $L_{2}$ norm.

Based on the Eq. (7), the specific formulas for the representative GLM models can be derived:

i) Linear regression: $A(\theta)=\frac{1}{2}\theta^{2}$, then

[TABLE]

where $\circ$ denotes the element-wise product and the $\left\|\boldsymbol{x}\circ(\boldsymbol{w}+c\boldsymbol{s})\right\|_{2}^{2}$ term can be decomposed into the summation of three components

[TABLE]

where $\boldsymbol{1}_{w_{j}\neq 0}[w_{j}]$ is an indicator function which satisfies $\boldsymbol{1}_{w_{j}\neq 0}[w_{j}]=\begin{cases}1&w_{j}\neq 0\\ 0&w_{j}=0\end{cases}$. This decomposition implies that Shakeout regularizer penalizes the combination of $L_{0}$-norm, $L_{1}$-norm and $L_{2}$-norm of the weights after scaling them with the square of corresponding features. The $L_{0}$ and $L_{1}$ regularization terms can lead to sparse weights.

ii) Logistic regression: $A(\theta)=\ln(1+\exp(\theta))$, then

[TABLE]

Fig. 3 illustrates the contour of Shakeout regularizer based on Eq. (9) in the 2D weight space. On the whole, the contour of Shakeout regularizer indicates that the regularizer combines $L_{0}$, $L_{1}$ and $L_{2}$ regularization terms. As $c$ goes to zero, the contour around $w=0$ becomes less sharper, which implies hyper-parameter $c$ relates to the strength of $L_{0}$ and $L_{1}$ components. When $c=0$, Shakeout degenerates to Dropout, the contour of which implies Dropout regularizer consists of $L_{2}$ regularization term.

The difference between Shakeout and Dropout regularizers is also illustrated in Fig. 2. We set $\tau=0.3$, $c=0.78$ for Shakeout, and $\tau=0.5$ for Dropout to make the bounds of the regularization effects of two regularizers the same. In this one dimension circumstance, the main difference is that at $w=0$ (see the enlarged snapshot for comparison), Shakeout regularizer is sharp and discontinuous while Dropout regularizer is smooth. Thus compared to Dropout, Shakeout may lead to much sparser weights of the model.

To simplify the analysis and prove the intuition we have observed in Fig. 3 about the properties of Shakeout regularizer, we quadratically approximate Shakeout regularizer of Eq. (7) by

[TABLE]

The $\left\|\boldsymbol{x}\circ(\boldsymbol{w}+c\boldsymbol{s})\right\|_{2}^{2}$, already shown in Eq. (8), consists of the combination of $L_{0}$, $L_{1}$, $L_{2}$ regularization terms. It tends to penalize the weight whose corresponding feature’s magnitude is large. Meanwhile, the weights whose corresponding features are always zeros are less penalized. The term $A^{{}^{\prime\prime}}(\theta)$ is proportional to the variance of prediction $y$ given $\boldsymbol{x}$ and $\boldsymbol{w}$. Penalizing $A^{{}^{\prime\prime}}(\theta)$ encourages the weights to move towards making the model be more ”confident” about its predication, i.e. be more discriminative.

Generally speaking, Shakeout regularizer adaptively combines $L_{0}$, $L_{1}$ and $L_{2}$ regularization terms, the property of which matches what we have observed in Fig. 3. It prefers penalizing the weights who have large magnitudes and encourages the weights to move towards making the model more discriminative. Moreover, the weights whose corresponding features are always zeros are less penalized. The $L_{0}$ and $L_{1}$ components can induce sparse weights.

Last but not the least, we want to emphasize that when $\tau=0$, the noise is eliminated and the model becomes a standard GLM. Moreover, Dropout can be viewed as the special case of Shakeout when $c=0$, and a higher value of $\tau$ means a stronger $L_{2}$ regularization effect imposed on the weights. Generally, when $\tau$ is fixed ($\tau\neq 0)$, a higher value of $c$ means a stronger effect of the $L_{0}$ and $L_{1}$ components imposed and leads to much sparser weights of the model. We will verify this property in our experiment section later.

3.2 Shakeout in Multilayer Neural Networks

It has been illustrated that Shakeout regularizes the weights in linear modules. Linear module is the basic component of multilayer neural networks. That is, the linear operations connect the outputs of two successive layers. Thus Shakeout is readily applicable to the training of multilayer neural networks.

Considering the forward computation from layer $l$ to layer $l+1$, for a fully-connected layer, the Shakeout forward computation is as follows

[TABLE]

[TABLE]

where $i$ denotes the index of the output unit of layer $l+1$, and $j$ denotes the index of the output unit of layer $l$. The output unit of a layer is represented by $x$. The weight of the connection between unit $x_{j}$ and unit $x^{{}^{\prime}}_{i}$ is represented as $W_{ij}$. The bias for the ${i}$-th unit is denoted by $b_{i}$. The $S_{ij}$ is the sign of corresponding weight $W_{ij}$. After Shakeout operation, the linear combination $u_{i}$ is sent to the activation function $f(\cdot)$ to obtain the corresponding output $x^{{}^{\prime}}_{i}$. Note that the weights $W_{ij}$ that connect to the same input unit $x_{j}$ are controlled by the same random variable $r_{j}$.

During back-propagation, we should compute the gradients with respect to each unit to propagate the error. In Shakeout, $\frac{\partial u_{i}}{\partial x_{j}}$ takes the form

[TABLE]

And the weights are updated following

[TABLE]

where $\frac{dS_{ij}}{dW_{ij}}$ represents the derivative of a sgn function. Because the sgn function is not continuous at zero and thus the derivative is not defined, we approximate this derivative with $\frac{d\tanh(W_{ij})}{dW_{ij}}$. Empirically we find that this approximation works well.

Note that the forward-backward computations with Shakeout can be easily extended to the convolutional layer. For a convolutional layer, the Shakeout feed-forward process can be formalized as

[TABLE]

[TABLE]

where $\mathbf{X}_{j}$ represents the $j$-th feature map. $\mathbf{R}_{j}$ is the $j$-th random mask which has the same spatial structure (i.e. the same height and width) as the corresponding feature map $\mathbf{X}_{j}$. $\mathbf{W}_{ij}$ denotes the kernel connecting $\mathbf{X}_{j}$ and $\mathbf{U}_{i}$. And $\mathbf{S}_{ij}$ is set as $\mathrm{sgn}(\mathbf{W}_{ij})$. The symbol

• denotes the convolution operation. And the symbol $\circ$ means element-wise product.

Correspondingly, during the back-propagation process, the gradient with respect to a unit of the layer on which Shakeout is applied takes the form

[TABLE]

where $(a,b)$ means the position of a unit in the output feature map of a layer, and $(a^{{}^{\prime}},b^{{}^{\prime}})$ represents the position of a weight in the corresponding kernel.

The weights are updated following

[TABLE]

4 Experiments

In this section, we report empirical evaluations of Shakeout in training deep neural networks on representative datasets. The experiments are performed on three kinds of image datasets: the hand-written image dataset MNIST [40], the CIFAR-10 image dataset [49] and the ImageNet-2012 dataset [50]. MNIST consists of 60,000+10,000 (training+test) 28$\times$28 images of hand-written digits. CIFAR-10 contains 50,000+10,000 (training+test) 32$\times$32 images of 10 object classes. ImageNet-2012 consists of 1,281,167+50,000+150,000 (training+validation+test) variable-resolution images of 1000 object classes. We first demonstrate that Shakeout leads to sparse models as our theoretical analysis implies under the unsupervised setting. Then we show that for the classification task, the sparse models have desirable generalization performances. Further, we illustrate the regularization effect of Shakeout on the weights in the classification task. Moreover, the effect of Shakeout on stabilizing the training processes of the deep architectures is demonstrated. Finally, we give some practical recommendations of Shakeout. All the experiments are implemented based on the modifications of Caffe library [51]. Our code is released on the github: https://github.com/kgl-prml/shakeout-for-caffe.

4.1 *Shakeout and Weight

Sparsity*

Since Shakeout implicitly imposes $L_{0}$ penalty and $L_{1}$ penalty of the weights, we expect the weights of neural networks learned by Shakeout contain more zeros than those learned by the standard back-propagation (BP) [52] or Dropout [13]. In this experiment, we employ an autoencoder model for the MNIST hand-written data, train the model using standard BP, Dropout and Shakeout, respectively, and compare the degree of sparsity of the weights of the learned encoders. For the purpose of demonstration, we employ the simple autoencoder with one hidden layer of 256 units; Dropout and Shakeout are applied on the input pixels.

To verify the regularization effect, we compare the weights of the four autoencoders trained under different settings which correspond to standard BP, Dropout ($\tau=0.5$) and Shakeout ($\tau=0.5$, $c=\{1,10\}$). All the training methods aim to produce hidden units which can capture good visual features of the handwritten digits. The statistical traits of these different resulting weights are shown in Fig. 4. Moreover, Fig. 5 shows the features captured by each hidden unit of the autoencoders.

As shown in the Fig. 4, the probability density of weights around the zero obtained by standard BP training is quite small compared to the one obtained either by Dropout or Shakeout. This indicates the strong regularization effect induced by Dropout and Shakeout. Furthermore, the sparsity level of weights obtained from training by Shakeout is much higher than the one obtained from training by Dropout. Using the same $\tau$, increasing $c$ makes the weights much sparser, which is consistent with the characteristics of $L_{0}$ penalty and $L_{1}$ penalty induced by Shakeout. Intuitively, we can find that due to the induced $L_{2}$ regularization, the distribution of weights obtained from training by the Dropout is like a Gaussian, while the one obtained from training by Shakeout is more like a Laplacian because of the additionally induced $L_{1}$ regularization. Fig. 5 shows that features captured by the hidden units via standard BP training are not directly interpretable, corresponding to insignificant variants in the training data. Both Dropout and Shakeout suppress irrelevant weights by their regularization effects, where Shakeout produces much sparser and more global features thanks to the combination of $L_{0}$, $L_{1}$ and $L_{2}$ regularization terms.

The autoencoder trained by Dropout or Shakeout can be viewed as the denosing autoencoder, where Dropout or Shakeout injects special kind of noise into the inputs. Under this unsupervised setting, the denoising criterion (i.e. minimizing the error between imaginary images reconstructed from the noisy inputs and the real images without noise) is to guide the learning of useful high level feature representations [11, 12]. To verify that Shakeout helps learn better feature representations, we adopt the hidden layer activations as features to train SVM classifiers, and the classification accuracies on test set for standard BP, Dropout and Shakeout are 95.34%, 96.41% and 96.48%, respectively. We can see that Shakeout leads to much sparser weights without defeating the main objective.

Gaussian Dropout has similar effect on the model training as standard Dropout [34], which multiplies the activation of each unit by a Gaussian variable with mean 1 and variance $\sigma^{2}$. The relationship between $\sigma^{2}$ and $\tau$ is that $\sigma^{2}=\frac{\tau}{1-\tau}$. The distribution of the weights trained by Gaussian Dropout ($\sigma^{2}=1$, i.e. $\tau=0.5$) is illustrated in Fig. 4. From Fig. 4, we find no notable statistical difference between two kinds of Dropout implementations which all exhibit a kind of $L_{2}$ regularization effect on the weights. The classification performances of SVM classifiers on test set based on the hidden layer activations as extracted features for both kinds of Dropout implementations are quite similar (i.e. $96.41\%$ and $96.43\%$ for standard and Gaussian Dropout respectively). Due to these observations, we conduct the following classification experiments using standard Dropout as a representative implementation (of Dropout) for comparison.

4.2 Classification Experiments

Sparse models often indicate lower complexity and better generalization performance [53, 26, 39, 54]. To verify the effect of $L_{0}$ and $L_{1}$ regularization terms induced by Shakeout on the model performance, we apply Shakeout, along with Dropout and standard BP, on training representative deep neural networks for classification tasks. In all of our classification experiments, the hyper-parameters $\tau$ and $c$ in Shakeout, and the hyper-parameter $\tau$ in Dropout are determined by validation.

4.2.1 MNIST

We train two different neural networks, a shallow fully-connected one and a deep convolutional one. For the fully-connected neural network, a big hidden layer size is adopted with its value at 4096. The non-linear activation unit adopted is the rectifier linear unit (ReLU). The deep convolutional neural network employed is based on the modifications of the LeNet [40], which contains two convolutional layers and two fully-connected layers. The detailed architecture information of this convolutional neural network is described in Tab. I.

We separate 10,000 training samples from original training dataset for validation. The results are shown in Tab. II and Tab. III. Dropout and Shakeout are applied on the hidden units of the fully-connected layer. The table compares the errors of the networks trained by standard back-propagation, Dropout and Shakeout. The mean and standard deviation of the classification errors are obtained by 5 runs of the experiment and are shown in percentage. We can see from the results that when the training data is not sufficient enough, due to over-fitting, all the models perform worse. However, the models trained by Dropout and Shakeout consistently perform better than the one trained by standard BP. Moreover, when the training data is scarce, Shakeout leads to superior model performance compared to the Dropout. Fig. 6 shows the results in a more intuitive way.

4.2.2 CIFAR-10

We use the simple convolutional network feature extractor described in cuda-convnet (layers-80sec.cfg) [55]. We apply Dropout and Shakeout on the first fully-connected layer. We call this architecture “AlexFastNet” for the convenience of description. In this experiment, 10,000 colour images are separated from the training dataset for validation and no data augmentation is utilized. The per-pixel mean computed over the training set is subtracted from each image. We first train for 100 epochs with an initial learning rate of 0.001 and then another 50 epochs with the learning rate of 0.0001. The mean and standard deviation of the classification errors are obtained by 5 runs of the experiment and are shown in percentage.

As shown in Tab. IV, the performances of models trained by Dropout and Shakeout are consistently superior to the one trained by standard BP. Furthermore, the model trained by Shakeout also outperforms the one trained by Dropout when the training data is scarce. Fig. 7 shows the results in a more intuitive way.

To test the performance of Shakeout on a much deeper architecture, we also conduct experiments based on the Wide Residual Network (WRN) [15]. The configuration of WRN adopted is WRN-16-4, which means WRN has 16 layers in total and the number of feature maps for the convolutional layer of each residual block is 4 times as the corresponding original one [1]. Because the complexity is much higher than that of “AlexFastNet”, the experiments are performed on relatively larger training sets with sizes of 15000, 40000, 50000. Dropout and Shakeout are applied on the second convolutional layer of each residual block, following the protocol in [15]. All the training starts from the same initial weights. Batch Normalization is applied the same way as [15] to promote the optimization. No data-augmentation or data pre-processing is adopted. All the other hyper-parameters other than $\tau$ and $c$ are set the same as [15]. The results are listed in Tab. V. For the training of CIFAR-10 with 50000 training samples, we adopt the same hyper-parameters as those chosen in the training with training set size at 40000. From Tab. V, we can arrive at the same conclusion as previous experiments, i.e. the performances of the models trained by Dropout and Shakeout are consistently superior to the one trained by standard BP. Moreover, Shakeout outperforms Dropout when the data is scarce.

4.2.3 Regularization Effect on the Weights

Shakeout is a different way to regularize the training process of deep neural networks from Dropout. For a GLM model, we have proved that the regularizer induced by Shakeout adaptively combines $L_{0}$, $L_{1}$ and $L_{1}$ regularization terms. In section 4.1, we have demonstrated that for a one-hidden layer autoencoder, it leads to much sparser weights of the model. In this section, we will illustrate the regularization effect of Shakeout on the weights in the classification task and make a comparison to that of Dropout.

The results shown in this section are mainly based on the experiments conducted on ImageNet-2012 dataset using the representative deep architecture: AlexNet [14]. For AlexNet, we apply Dropout or Shakeout on layers FC7 and FC8 which are the last two fully-connected layers. We train the model from the scratch and obtain the comparable classification performances on validation set for Shakeout (top-1 error: 42.88%; top-5 error: 19.85%) and Dropout (top-1 error: 42.99%; top-5 error: 19.60%). The model is trained based on the same hyper-parameter settings provided by Shelhamer in Caffe [51] other than the hyper-parameters $\tau$ and $c$ for Shakeout. The initial weights for training by Dropout and Shakeout are kept the same.

Fig. 8 illustrates the distributions of the magnitude of weight resulted by Shakeout and Dropout. It can be seen that the weights learned by Shakeout are much sparser than those learned by Dropout, due to the implicitly induced $L_{0}$ and $L_{1}$ components.

The regularizer induced by Shakeout not only contains $L_{0}$ and $L_{1}$ regularization terms but also contains $L_{2}$ regularization term, the combination of which is expected to discard a group of weights simultaneously. In Fig. 9, we use the maximum magnitude of the weights connected to one input unit of a layer to represent the importance of that unit for the subsequent output units. From Fig. 9, it can be seen that for Shakeout, the units can be approximately separated into two groups according to the maximum magnitudes of the connected weights and the group around zero can be discarded, whereas for Dropout, the units are concentrated. This implies that compared to Dropout which may encourage a “distributed code” for the features captured by the units of a layer, Shakeout tends to discard the useless features (or units) and award the important ones. This experiment result verifies the regularization properties of Shakeout and Dropout further.

As known to us, $L_{0}$ and $L_{1}$ regularization terms are related to performing feature selection [56, 57]. For a deep architecture, it is expected to obtain a set of weights using Shakeout suitable for reflecting the importance of connections between units. We perform the following experiment to verify this effect. After a model is trained, for the layer on which Dropout or Shakeout is applied, we sort the magnitudes of the weights increasingly. Then we prune the first $m\%$ of the sorted weights and evaluate the performance of the pruned model again. The pruning ratio $m$ goes from 0 to 1. We calculate the relative accuracy loss (we write $R.A.L$ for simplification) at each pruning ratio $m^{{}^{\prime}}$ as

[TABLE]

Fig. 10 shows the $R.A.L$ curves for Dropout and Shakeout based on the AlexNet model on ImageNet-2012 dataset. The models trained by Dropout and Shakeout are under the optimal hyper-parameter settings. Apparently, the relative accuracy loss for Dropout is more severe than that for Shakeout. For example, the largest margin of the relative accuracy losses between Dropout and Shakeout is $22.50\%$, which occurs at the weight pruning ratio $m=96\%$. This result proves that considering the trained weights in reflecting the importance of connections, Shakeout is much better than Dropout, which benefits from the implicitly induced $L_{0}$ and $L_{1}$ regularization effect.

This kind of property is useful for the popular compression task in deep learning area which aims to cut the connections or throw units of a deep neural network to a maximum extent without obvious loss of accuracy. The above experiments illustrate that Shakeout can play a considerable role in selecting important connections, which is meaningful for promoting the performance of a compression task. This is a potential subject for the future research.

4.3 Stabilization Effect on the Training Process

In both research and production, it is always desirable to have a level of certainty about how a model’s fitness to the data improves over optimization iterations, namely, to have a stable training process. In this section, we show that Shakeout helps reduce fluctuation in the improvement of model fitness during training.

The first experiment is on the family of Generative Adversarial Networks (GANs) [58], which is known to be instable in the training stage [59, 60, 61]. The purpose of the following tests is to demonstrate the Shakeout’s capability of stabilizing the training process of neural networks in a general sense. GAN plays a min-max game between the generator $G$ and the discriminator $D$ over the expected log-likelihood of real data $\boldsymbol{x}$ and imaginary data $\hat{\boldsymbol{x}}=G(\boldsymbol{z})$ where $\boldsymbol{z}$ represents the random input

[TABLE]

The architecture that we adopt is DCGAN [59]. The numbers of feature maps of the deconvolutional layers in the generator are 1024, 64 and 1 respectively, with the corresponding spatial sizes 7$\times$7, 14$\times$14 and 28$\times$28. We train DCGANs on MNIST dataset using standard BP, Dropout and Shakeout. We follow the same experiment protocol described in [59] except for adopting Dropout or Shakeout on all layers of the discriminator. The values of $-V(D,G)$ during training are illustrated in Fig. 11. It can be seen that $-V(D,G)$ during training by standard BP oscillates greatly, while for Dropout and Shakeout, the training processes are much steadier. Compared with Dropout, the training process by Shakeout has fewer spikes and is smoother. Fig. 12 demonstrates the minimum and maximum values of $-V(D,G)$ within fixed length intervals moving from the start to the end of the training by standard BP, Dropout and Shakeout. It can be seen that the gaps between the minimum and maximum values of $-V(D,G)$ trained by Dropout and Shakeout are much smaller than that trained by standard BP, while that by Shakeout is the smallest, which implies the stability of the training process by Shakeout is the best.

The second experiment is based on Wide Residual Network architecture to perform the classification task. In the classification task, generalization performance is the main focus and thus, we compare the validation errors during the training processes by Dropout and Shakeout. Fig. 13 demonstrates the validation error as a function of the training epoch for Dropout and Shakeout on CIFAR-10 with 40000 training examples. The architecture adopted is WRN-16-4. The experiment settings are the same as those described in Section 4.2.2. Considering the generalization performance, the learning rate schedule adopted is the one optimized through validation to make the models obtain the best generalization performances. Under this schedule, we find that the validation error temporarily increases when lowering the learning rate at the early stage of training, which has been repeatedly observed by [15]. Nevertheless, it can be seen from Fig. 13 that the extent of error increase is less severe for Shakeout than Dropout. Moreover, Shakeout recovers much faster than Dropout does. At the final stage, both of the validation errors steadily decrease. Shakeout obtains comparable or even superior generalization performance to Dropout. In a word, Shakeout significantly stabilizes the entire training process with superior generalization performance.

4.4 Practical Recommendations

Selection of Hyper-parameters The most practical and popular way to perform hyper-parameter selection is to partition the training data into a training set and a validation set to evaluate the classification performance of different hyper-parameters on it. Due to the expensive cost of time for training a deep neural network, cross-validation is barely adopted. There exist many hyper-parameter selection methods in the domain of deep learning, such as the grid search, random search [62], Bayesian optimization methods [63], gradient-based hyper-parameter Optimization [64], etc. For applying Shakeout on a deep neural network, we need to decide two hyper-parameters $\tau$ and $c$. From the regularization perspective, we need to decide the most suitable strength of regularization effect to obtain an optimal trade-off between model bias and variance. We have pointed out that in a unified framework, Dropout is a special case of Shakeout when Shakeout hyper-parameter $c$ is set to zero. Empirically we find that the optimal $\tau$ for Shakeout is not higher than that for Dropout. After determining the optimal $\tau$, keeping the order of magnitude of hyper parameter $c$ the same as $\sqrt{\frac{1}{N}}$ ($N$ represents the number of training samples) is an effective choice. If you want to obtain a model with much sparser weights but meanwhile with superior or comparable generalization performance to Dropout, a relatively lower $\tau$ and larger $c$ for Shakeout always works.

Shakeout combined with Batch Normalization Batch Normalization [65] is the widely-adopted technique to promote the optimization of the training process for a deep neural network. In practice, combining Shakeout with Batch Normalization to train a deep architecture is a good choice. For example, we observe that the training of WRN-16-4 model on CIFAR-10 is slow to converge without using Batch Normalization in the training. Moreover, the generalization performance on the test set for Shakeout combined with Batch Normalization always outperforms that for standard BP with Batch Normalization consistently for quite a large margin, as illustrated in Tab. V. These results imply the important role of Shakeout in reducing over-fitting of a deep neural network.

5 Conclusion

We have proposed Shakeout, which is a new regularized training approach for deep neural networks. The regularizer induced by Shakeout is proved to adaptively combine $L_{0}$, $L_{1}$ and $L_{2}$ regularization terms. Empirically we find that

1. Compared to Dropout, Shakeout can afford much larger models. Or to say, when the data is scarce, Shakeout outperforms Dropout with a large margin.

2. Shakeout can obtain much sparser weights than Dropout with superior or comparable generalization performance of the model. While for Dropout, if one wants to obtain the same level of sparsity as that obtained by Shakeout, the model may bear a significant loss of accuracy.

3. Some deep architectures in nature may result in the instability of the training process, such as GANs, however, Shakeout can reduce this instability effectively.

In future, we want to put emphasis on the inductive bias of Shakeout and attempt to apply Shakeout to the compression task.

Acknowledgments

This research is supported by Australian Research Council Projects (No. FT-130101457, DP-140102164 and LP-150100671).