Compressed Learning of Deep Neural Networks for OpenCL-Capable Embedded Systems

TL;DR

This paper introduces a novel model compression framework for deep neural networks that enables efficient training and inference on embedded systems using OpenCL, leveraging sparse representations learned during training.

Contribution

It presents a versatile compression method that does not require pre-trained models and improves effectiveness using proximal algorithms and debiasing techniques.

Findings

Achieves minimal models suitable for embedded devices

Uses sparse coding with proximal algorithms for better compression

Does not require pre-trained models for compression

Abstract

Deep neural networks (DNNs) have been quite successful in solving many complex learning problems. However, DNNs tend to have a large number of learning parameters, leading to a large memory and computation requirement. In this paper, we propose a model compression framework for efficient training and inference of deep neural networks on embedded systems. Our framework provides data structures and kernels for OpenCL-based parallel forward and backward computation in a compressed form. In particular, our method learns sparse representations of parameters using $\ell_1$-based sparse coding while training, storing them in compressed sparse matrices. Unlike the previous works, our method does not require a pre-trained model as an input and therefore can be more versatile for different application environments. Even though the use of $\ell_1$-based sparse coding for model compression is not new, we show that it can be far more effective than previously reported when we use proximal point algorithms and the technique of debiasing. Our experiments show that our method can produce minimal learning models suitable for small embedded devices.

Tables

Table 1. Table 1: Summary of network compression results. (Pru: network pruning, SpC: our method)

Network		Lenet-5	AlexNet	VGGNet	ResNet-32
Data		MNIST	CIFAR-10	CIFAR-10	CIFAR-10
Ref. Accuracy		$98.61\%$	$78.61\%$	$84.88\%$	$90.05\%$
Pru	Accuracy	$37.01\%$	$11.24\%$	$10.00\%$	$89.76\%$
	Compression Rate	$0.97(29\times )$	$0.91(10\times )$	$0.94(15\times )$	$0.21(1\times )$
Pru (Retrain)	Accuracy	$98.19\%$	$42.67\%$	$10.00\%$	$88.16\%$
	Compression Rate	$0.97(33\times )$	$0.97(35\times )$	$0.98(51\times )$	$0.88(8\times )$
SpC	Accuracy	$97.78\%$	$80.93\%$	$85.53\%$	$90.22\%$
	Compression Rate	$0.97(32\times )$	$0.91(10\times )$	$0.94(17\times )$	$0.21(1\times )$
SpC (Retrain)	Accuracy	$98.29\%$	$78.84\%$	$84.63\%$	$89.22\%$
	Compression Rate	$0.97(35\times )$	$0.98(47\times )$	$0.98(53\times )$	$0.88(8\times )$

Table 2. Table 2: Comparison between our method SpC and the state-of-the-art MM. For MM, we used the hyperparameters and the best pretrained models as a starting point following the original paper  (Carreira-Perpiñán and Idelbayev, 2018 ) . Note that our method SpC starts from random weights and does not require complicate control of the auxiliary parameter μ 𝜇 \mu .

Network (Data)	Lenet-5 (MNIST)		ResNet32 (CIFAR10)
Method	SpC	MM	SpC	MM
Pretrained Model	-	Required (Test Acc = 99.1)	-	Required (Test Acc = 92.28)
Solver	Prox-Adam	SGD with Momentum	Prox-Adam	Nesterov
Aux. Parameter ($\mu$)	-	$9.76\times {10}^{-5}$ ($\times 1.1$ per 4k iter)	-	${10}^{-3}$($\times 1.1$ per 2k iter)
Accuracy	97.25%	97.65%	89.22%	92.37%
Compression Rate	0.98	0.98	0.88	0.85

Table 3. Table 3: Inference speedups by model compression (Lenet-5, MNIST).

GPU	NVIDIA GTX 1080 TI		ARM Mali-T860
Compression	Yes	No	Yes	No
Model Size	148KB	5.0MB	148KB	5.0MB
Inference Time	8572 ms	16,977 ms	506,067 ms	606,699 ms
Speed up	2$\times$		1.2$\times$

Table 4. Table 4: Layer-wise compression of lenet-5 (MNIST dataset). NNZ stands for number of non-zeros.

Methods	SpC		SpC(Retrain)
Layers	NNZ/Total Weights	Compression Rate	NNZ/Total Weights	Compression Rate
conv1	158/500	68.40% (3$\times$)	142/500	71.60% (3$\times$)
conv2	2101/25,000	91.60% (11$\times$)	1750/25,000	93.00% (14$\times$)
fc1	10,804/400,000	97.30% (37$\times$)	10,045/400,000	97.49% (39$\times$)
fc2	270/5000	94.60% (18$\times$)	280/5000	94.40% (17$\times$)
Total	13,333/430,500	96.90% (32$\times$)	12,217/430,500	97.16% (35$\times$)
Test Accuracy	0.9778 @ $\lambda$ = 1.26	Ref: 0.9861	0.9829 @ $\lambda$ = 1.28	Ref: 0.9861

Table 5. Table 5: Layer-wise compression of AlexNet (CIFAR-10 dataset).

Methods	SpC		SpC(Retrain)
Layers	NNZ / Total Weights	Compression Rate	NNZ / Total Weights	Compression Rate
conv1	3922/7200	45.53% (1$\times$)	2054/7200	71.47% (3$\times$)
conv2	76,321/307,200	75.16% (4$\times$)	19,464/307,200	93.66% (15$\times$)
conv3	153,921/884,736	82.60% (5$\times$)	27,757/88,4736	96.86% (31$\times$)
conv4	153,000/663,552	76.94% (4$\times$)	21,485/663,552	96.76% (30$\times$)
conv5	516,47/442,368	88.32% (8$\times$)	15,690/442,368	96.45% (28$\times$)
fc1	179,344/4,194,304	95.72% (23$\times$)	52,429/4,194,304	98.75% (79$\times$)
fc2	84,495/1,048,576	91.94% (12$\times$)	18,841/1,048,576	98.20% (55$\times$)
fc3	3701/10,240	63.86% (2$\times$)	2329/10,240	77.26% (4$\times$)
Total	706,351/7,558,176	90.65% (10$\times$)	160,049/7,558,176	97.88% (47$\times$)
Test Accuracy	0.8093 @ $\lambda$ = 1.03	Ref: 0.7861	0.7884 @ $\lambda$ = 1.06	Ref: 0.7861

Table 6. Table 6: Layer-wise compression of VGGNet (CIFAR-10 dataset).

Methods	SpC		SpC(Retrain)
Layers	NNZ/Total Weights	Compression Rate	NNZ/Total Weights	Compression Rate
conv1-1	1160/1728	32.87% (1$\times$)	757/1728	56.19% (2$\times$)
conv1-2	18,904/36,864	48.72% (1$\times$)	9389/36,864	74.53% (3$\times$)
conv2-1	47,497/73,728	35.58% (1$\times$)	20943/73,728	71.59% (3$\times$)
conv2-2	87,314/147,456	40.79% (1$\times$)	30,040/147,456	79.63% (4$\times$)
conv3-1	133,402/294,912	54.77% (2$\times$)	40,039/294,912	86.42% (7$\times$)

Table 7. Table 6: Cont .

Methods	SpC		SpC(Retrain)
Layers	NNZ/Total Weights	Compression Rate	NNZ/Total Weights	Compression Rate
conv3-2	120,094/589,824	79.64% (4$\times$)	30,997/589,824	94.74% (19$\times$)
conv3-3	94,612/589,824	83.96% (6$\times$)	16,071/589,824	97.28% (36$\times$)
conv4-1	164,660/1,179,648	86.04% (7$\times$)	20,322/1,179,648	98.28% (58$\times$)
conv4-2	133,944/2,359,296	94.32% (17$\times$)	22,145/2,359,296	99.06% (106$\times$)
conv4-3	59,355/2,359,296	97.48% (39$\times$)	28,173/2,359,296	98.81% (83$\times$)
conv5-1	16,749/2,359,296	99.29% (140$\times$)	21,349/2,359,296	99.10% (110$\times$)
conv5-2	10,769/2,359,296	99.54% (219$\times$)	30,008/2,359,296	98.73% (78$\times$)
conv5-3	10,987/2,359,296	99.53% (214$\times$)	24,027/2,359,296	98.98% (98$\times$)
fc1	4176/524,288	99.20% (125$\times$)	6072/524,288	98.84% (86$\times$)
fc2	5915/1,048,576	99.44% (177$\times$)	4007/1,048,576	99.62% (261$\times$)
fc3	508/10,240	95.04% (20$\times$)	223/10,240	97.82% (45$\times$)
Total	910,046/16,293,568	94.41% (17$\times$)	304,562/16,293,568	98.13% (53$\times$)
Test Accuracy	0.8553 @ $\lambda$ = 1.02	Ref: 0.8488	0.8463 @ $\lambda$ = 1.04	Ref: 0.8488

Table 8. Table 7: Layer-wise compression of ResNet32 (CIFAR-10 dataset).

Methods	SpC		SpC(Retrain)
Layers	NNZ / Total Weights	Compression Rate	NNZ / Total Weights	Compression Rate
conv1	379/432	12.27% (1$\times$)	139/432	67.82% (3$\times$)
conv1-1-1	1844/2304	19.97% (1$\times$)	327/2304	85.81% (7$\times$)
conv1-1-2	1870/2304	18.84% (1$\times$)	337/2304	85.37% (6$\times$)
conv1-2-1	1874/2304	18.66% (1$\times$)	334/2304	85.50% (6$\times$)
conv1-2-2	1873/2304	18.71% (1$\times$)	341/2304	85.20% (6$\times$)
conv1-3-1	1847/2304	19.84% (1$\times$)	330/2304	85.68% (6$\times$)
conv1-3-2	1872/2304	18.75% (1$\times$)	322/2304	86.02% (7$\times$)
conv1-4-1	1874/2304	18.66% (1$\times$)	363/2304	84.24% (6$\times$)
conv1-4-2	1852/2304	19.62% (1$\times$)	344/2304	85.07% (6$\times$)
conv1-5-1	1859/2304	19.31% (1$\times$)	355/2304	84.59% (6$\times$)
conv1-5-2	1835/2304	20.36% (1$\times$)	326/2304	85.85% (7$\times$)
conv2-1-1	3700/4608	19.70% (1$\times$)	666/4608	85.55% (6$\times$)
conv2-1-2	7316/9216	20.62% (1$\times$)	1108/9216	87.98% (8$\times$)
conv2-1-proj	467/512	8.79% (1$\times$)	225/512	56.05% (2$\times$)
conv2-2-1	7292/9216	20.88% (1$\times$)	1191/9216	87.08% (7$\times$)
conv2-2-2	7325/9216	20.52% (1$\times$)	1160/9216	87.41% (7$\times$)
conv2-3-1	7394/9216	19.77% (1$\times$)	1198/9216	87.00% (7$\times$)
conv2-3-2	7371/9216	20.02% (1$\times$)	1160/9216	87.41% (7$\times$)
conv2-4-1	7323/9216	20.54% (1$\times$)	1222/9216	86.74% (7$\times$)
conv2-4-2	7368/9216	20.05% (1$\times$)	1200/9216	86.98% (7$\times$)
conv2-5-1	7265/9216	21.17% (1$\times$)	1223/9216	86.73% (7$\times$)
conv2-5-2	7303/9216	20.76% (1$\times$)	1179/9216	87.21% (7$\times$)
conv3-1-1	14,757/18,432	19.94% (1$\times$)	2411/18,432	86.92% (7$\times$)
conv3-1-2	29,393/36,864	20.27% (1$\times$)	4281/36,864	88.39% (8$\times$)
conv3-1-proj	1815/2048	11.38% (1$\times$)	638/2048	68.85% (3$\times$)
conv3-2-1	29,423/36,864	20.19% (1$\times$)	4399/36,864	88.07% (8$\times$)
conv3-2-2	29,372/36,864	20.32% (1$\times$)	4442/36,864	87.95% (8$\times$)
conv3-3-1	29,332/36,864	20.43% (1$\times$)	4427/36,864	87.99% (8$\times$)
conv3-3-2	29,244/36,864	20.67% (1$\times$)	4392/36,864	88.09% (8$\times$)
conv3-4-1	29,264/36,864	20.62% (1$\times$)	4450/36,864	87.93% (8$\times$)
conv3-4-2	28,954/36,864	21.46% (1$\times$)	4413/36,864	88.03% (8$\times$)
conv3-5-1	28,770/36,864	21.96% (1$\times$)	4469/36,864	87.88% (8$\times$)
conv3-5-2	28,455/36,864	22.81% (1$\times$)	4381/36,864	88.12% (8$\times$)
fc1	570/640	10.94% (1$\times$)	298/640	53.44% (2$\times$)
Total	368452/464432	20.67% (1$\times$)	58051/464432	87.50% (8$\times$)
Test Accuracy	0.9022 @ $\lambda$ = 1.001	Ref: 0.9005	0.8922 @ $\lambda$ = 1.005	Ref: 0.9005

Equations

$$w^{\ast }\in \underset{w\in {\mathbb{R}}^p}{\mathrm{argmin}}\frac{1}{n}\sum _{i=1}^n\ell (w;x_i,y_i).$$

$$w^{\ast }\in \underset{w\in {\mathbb{R}}^p}{\mathrm{argmin}}\frac{1}{n}\sum _{i=1}^n\ell (w;x_i,y_i)+\Psi (w).$$

$$g(w):=g_B(w)+\nabla \Psi (w),$$

$${\text{prox}}_{\Psi }(z):=\underset{w\in {\mathbb{R}}^p}{\mathrm{argmin}}\frac{1}{2}\Vert w-z{\Vert }_2^2+\Psi (w).$$

$$[{\text{prox}}_{\Psi }(z){]}_i=\text{sgn}(z_i)\max \{|z_i|-\lambda ,0\},i=1,\dots ,p.$$

$$w^{k+1}\leftarrow {\text{prox}}_{\eta \Psi }\left(w^k-\eta G(w^k)\right).$$

$$G(w^k):=\frac{1}{n}\sum _{i=1}^n\nabla \ell (w^k;x_i,y_i).$$

$$w^{k+1}\leftarrow {\text{prox}}_{\eta \Psi }\left(w^k-\eta g_B(w^k)\right).$$

$$A=\left[\begin{array}{cccc}1& 7& 0& 0\\ 0& 2& 8& 0\\ 5& 0& 3& 9\\ 0& 6& 0& 4\end{array}\right]$$

$$\text{data}=\left[\begin{array}{ccc}\ast & 1& 7\\ \ast & 2& 8\\ 5& 3& 9\\ 6& 4& \ast \end{array}\right],\text{offsets}=[-201]$$

$$\text{data}=\left[\begin{array}{ccc}1& 7& \ast \\ 2& 8& \ast \\ 5& 3& 9\\ 6& 4& \ast \end{array}\right],\text{indices}=\left[\begin{array}{ccc}0& 1& \ast \\ 1& 2& \ast \\ 0& 2& 3\\ 1& 3& \ast \end{array}\right]$$

$$=[02479]$$

$$=[011202313]$$

$$=[172853964]$$

$$=[001122233]$$

$$=[011202313]$$

$$=[172853964]$$

$$:X_T\leftarrow X_B{W}^{\prime }\text{(dense×compressed′)},$$

$$:\frac{\partial Loss}{\partial X_B}\leftarrow \frac{\partial Loss}{\partial X_T}W\text{(dense×compressed)}.\text{vskip}6.0ptplus2.0ptminus2.0pt$$

$$\underset{w\in {\mathbb{R}}^p,\theta \in {\mathbb{R}}^p}{\min }L(w)+\alpha \Psi (\theta ),\text{subject to}w=\theta ,$$

$$\mathcal{L}(w,\theta ,\lambda ;\mu )=L(w)+\frac{\mu }{2}\Vert w-\theta {\Vert }^2-{\lambda }^T(w-\theta )+\Psi (\theta ).$$

Full text

\pubvolume

9 \issuenum8 \articlenumber0

\historyReceived: 21 March 2019; Accepted: 16 April 2019; Published: date \updatesyes

\TitleCompressed Learning of Deep Neural Networks for OpenCL-Capable Embedded Systems \AuthorSangkyun Lee *1,**\orcidA and Jeonghyun Lee 2 \AuthorNamesSangkyun Lee and Jeonghyun Lee

\corresCorrespondence: sangkyun@hanyang.ac.kr; Tel.: +82-31-400-1039

1 Introduction

Modern deep neural networks (DNNs) tend to have growing numbers of parameters, which are often unavoidable to solve complex learning problems in computer vision (Krizhevsky et al., 2012), speech recognition (Graves and Schmidhuber, 2005), and natural language processing problems (Collobert et al., 2011). For example, the number of parameters of convolutional neural networks (CNNs) for computer vision tasks has grown. Compared to the first CNN, Lenet-5 (Lecun et al., 1998), which contains less than 1M parameters, recent networks such as AlexNet (Krizhevsky et al., 2012) (60M) and VGG16 (Simonyan and Zisserman, 2015) (138M) consist of a much larger number of parameters.

Some networks have been optimized for their size, for example, GoogleNet (also known as Inception v1) (Szegedy et al., 2015), the winning method of ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2014, only has 6.7M parameters, a surprisingly smaller number compared to the previous winners (AlexNet (Krizhevsky et al., 2012) in 2012, ZFNet (Zeiler and Fergus, 2014) in 2013). However, new versions of GoogleNet such as BN-Inception (Ioffe and Szegedy, 2015), Inception v2 and v3 Szegedy et al. (2016), and Inception v4 Szegedy et al. (2017) tend to become larger for better prediction accuracy, for example, Inception v4 contains about 80M parameters. ResNet-152 (He et al., 2016), the winner of ILSVRC 2015 defeating Inception v3, contains 60M parameters. Other examples include Deepfaces (Taigman et al., 2014) (120M parameters) and DNNs on high-performance computing systems (Coates et al., 2013) (10B parameters).

Large numbers of learning parameters, however, make it preventive to learn and apply machine learning models on small devices, e.g., smartphones, embedded computers, and wearable devices, where memory, computation, and energy consumption can be restricted. In addition, a large number of parameters may lead to overfitting, a phenomenon in machine learning that a complex and large machine learning model fits training data too much and its generalization performance on future data decreases as a result (Lawrence et al., 1997). Therefore, compressing large machine learning models is an essential consideration for the training and the deployment of them in application environments such as edge computing (Buyya et al., 2009) with small computation devices.

1.1 Related Work

The main idea of model compression is to reduce the effective number of parameters to be stored and computed. One of the classical ideas for DNN model compression is to use a low-rank approximation of weight matrices. This technique has been applied successfully in scene text character recognition (Jaderberg et al., 2014), and later improved and extended for larger DNNs (Denton et al., 2014; Ioannou et al., 2016; Tai et al., 2016), reducing memory consumption and computational cost. However, these approaches work on fixed network architecture, requiring reiterations of decomposition, training, fine-tuning, and cross-validation.

1.1.1 Network Pruning

For the model compression of DNNs, network pruning is probably the most popular approach, which tries to remove irrelevant neural connections associated with weight values below a certain threshold. In biased weight decay (Hanson and Pratt, 1989), hyperbolic and exponential biases were introduced to the pruning objective, where brain damage (LeCun et al., 1990) and brain surgeon (Hassibi and Stork, 1993) used information from the Hessian of the loss function.

More recently, Han et al. (2015) suggested a network pruning approach with retraining, which performs weight training, network pruning, and then another round of weight training (retraining). The final retraining is done only for the connections not removed during the pruning step, in order to eliminate statistical bias caused by network pruning. The authors reported that pruning with $\ell_{2}$ regularization and retraining was the most successful regarding prediction accuracy, compared to $\ell_{1}$ and $\ell_{2}$ regularization alternatives with and without retraining. This method has further developed into deep compression (Han et al., 2016), which adds two extra steps after pruning: weight quantization and the Huffman encoding. This method has been followed-up with more emphasis on compressing weights in convolutional layers, using structured sparsity regularization to induce structured sparsity in rows, columns, channels, and filters (Wen et al., 2016; Lebedev and Lempitsky, 2016).

1.1.2 Network Pruning as Optimization

Since network pruning can be achieved by adding a regularization term in the training objective function, model compression by network pruning can be considered in terms of optimization problems. For example, stochastic optimization algorithms for $\ell_{1}$ regularization problems (Shalev-Shwartz and Tewari, 2009; Duchi and Singer, 2009) have been applied to CNN model compression (Zhou et al., 2016). The forward-backward splitting algorithms in these works are closely related to our proximal point algorithms (Parikh and Boyd, 2014) to be discussed later, but only classical stochastic gradient descent has been considered in the previous work, whereas we consider more advanced training algorithms for neural networks such as RMSProp (Hinton, 2014) and ADAM (Kingma and Ba, 2015).

The state-of-the-art network pruning approach (Carreira-Perpiñán and Idelbayev, 2018) considers reforming the network pruning into constrained optimization problems, solving them using the augmented Lagrangian method, also known as the method of multipliers (Hestenes, 1969; Powell, 1969). Our proposed method also considers the penalized optimization problem as in this approach, with an important distinction that we use proximal point algorithms (Parikh and Boyd, 2014) which comes with clear benefits: lower memory consumption and faster convergence, as we show later.

Other works in model compression include guiding compression with side information such as network latency provided in forms of optimization constraints (Chen et al., 2018), designing network architecture using reinforcement learning (e.g., NAS (Zoph and Le, 2016) and NASNet (Zoph et al., 2017)), and tuning parameters that trade-off model size and prediction accuracy automatically (He et al., 2018). We refer to Cheng et al. (2018) for a more detailed survey of DNN model compression.

1.2 Contribution

Even though there are quite a few works on model compression for deep neural networks, we found that they may not be well suited for learning and inference with DNNs in small embedded systems:

• •

Full model training is required prior to model compression: in network pruning (Han et al., 2015, 2016), including the state-of-the-art method (Carreira-Perpiñán and Idelbayev, 2018), and low-rank approximation (Denton et al., 2014; Ioannou et al., 2016; Jaderberg et al., 2014; Tai et al., 2016), full models have to be trained first, which would be too burdensome to run on small computing devices. In addition, compression must be followed by retraining or fine-tuning steps with a substantial number of training iterations, since, otherwise, the compressed models tend to show impractically low prediction accuracy, as we show in our experiments.

• •

Platforms are restricted: existing approaches (Han et al., 2015, 2016; Wen et al., 2016) have been implemented and evaluated mainly on CPUs because sparse weight matrices produced by model compression typically have irregular nonzero patterns and therefore are not suitable for GPU computation. In Wen et al. (2016), some evaluations have been performed on GPUs, but it was implemented with a proprietary sparse matrix library (cuSPARSE) only available on the NVIDIA platforms (Santa Clara, CA, USA). He et al. (2018) considered auto-tuning of model compression experimented on NVIDIA GPU and a mobile CPU on Google Pixel-1 (Mountain View, CA, USA), yet without discussion on how to implement them efficiently on embedded GPUs.

In this paper, we provide a model compression method for deep neural networks based on $\ell_{1}$-regularized optimization, often referred to as sparse coding (Lee et al., 2007; Needell and Tropp, 2009). Application of $\ell_{1}$-based regularization for DNNs is not entirely new; however, we discuss that it can be far more effective than previously reported. In particular, we apply $\ell_{1}$-based sparse coding with (i) proximal operators, which induce explicit sparsity in learning weights during training. In addition, it is followed by a (ii) debiasing step, which tries to reduce estimation bias due to sparse coding: we discuss that this technique allows us to compress DNNs further without sacrificing prediction accuracy much. While inducing sparsity on learning weights, we also store sparse weights in (iii) compressed data structures, being able to compute forward and backward passes with compressed weights on any (iv) OpenCL-capable devices, e.g., mobile GPUs such as ARM Mali-T880 (Cambridge, UK).

2 Method

In this section, we describe the details of our method in which network pruning is considered as an $\ell_{1}$-regularized optimization problem, i.e, sparse coding, where we aim to find a sparse representation of observations, sometimes also looking for good basis functions for such a representation (this particular task is called dictionary learning (Bao et al., 2016)). The idea of sparse coding has been applied in various fields of research, i.e., sparse regression in statistics (e.g., Tibshirani (1996)), compressed sensing in signal processing (e.g., Candes and Tao (2007, 2005); Donoho (2006)), matrix completion in machine learning (e.g., Candes and Plan (2010)), just to name a few.

In our approach, we apply sparse coding for model compression, to find a sparse optimal weight parameters containing as many zero values as possible whenever the associated neural connections in a deep neural network are irrelevant for making an accurate prediction.

2.1 Sparse Coding in Training

To describe our method formally, let us suppose that we train a deep neural network with $n$ examples, $\{(x_{i},y_{i})\}_{i=1}^{n}$, where $x_{i}\in\mathbb{R}^{d}$ is an input vector and $y_{i}\in\mathbb{R}$ (regression) or $y_{i}\in\{1,2,\dots,K\}$ (classification) is the corresponding outcome. In training, we obtain the optimal weight $w^{*}\in\mathbb{R}^{p}$ by solving the following optimization problem:

[TABLE]

Here, $\ell(w;x_{i},y_{i})$ is the loss function describing the gap between the prediction made by the neural net and the desired outcome. In sparse coding, we add a regularizer $\Psi(w)$ to the objective function and solve a modified training problem:

[TABLE]

We use the $\ell_{1}$-norm $\Psi(w)=\lambda\|w\|_{1}$ for our method in particular, where $\lambda>0$ is a hyperparameter that controls the rate of model compression (the larger $\lambda$ is, the resulting model will be more compressed as more weight values will set to the zero value).

Note that the use of $\ell_{1}$-regularizer by itself does not automatically lead to sparse coding. We need an explicit mechanism that sets weight values to the zero value while solving Equation (1). The mechanism is called a proximal operator.

2.2 Proximal Operators

When we solve the training problem (1) for a deep neural network with a large dataset, the most popular optimization algorithms include stochastic subgradient descent (Nemirovski et al., 2009) and its variants such as AdaGrad (Duchi et al., 2011), RMSProp (Hinton, 2014), and ADAM (Kingma and Ba, 2015). These methods use small subsamples of training examples $B\subset\{1,\dots,n\}$, called minibatches, to construct stochastic subgradients in forms of:

[TABLE]

where $g_{B}(w):=\frac{1}{|B|}\sum_{i\in B}\nabla\ell(w;x_{i},y_{i})$ and $\nabla\Psi(w)$ is a subgradient of $\Psi(w)$. If we use the subgradient $g(w)$ which includes the part $\nabla\Psi(w)$ from the regularizer for updating learning parameters $w$, it is unlikely that any updated weight value will be precisely the zero value, even though some values might be close to zero.

Instead, we consider a mechanism to set the weight values to the exact zero value whenever necessary, by means of the so-called proximal operator of a convex function $\Psi(\cdot)$. Given a vector $z\in\mathbb{R}^{p}$, the proximal operator of $z$ with respect to $\Psi$ is defined as

[TABLE]

When $\Psi(w)=\lambda\|w\|_{1}$, the regularizer we use for sparse coding, the proximal operator can be computed in a closed form independently for each component,

[TABLE]

This operator is also known as the soft-thresholding operator (Lee and Wright, 2011).

2.3 Optimization Algorithm

When the number of training examples $n$ is not too large, say $n\sim 10k$, we can solve the training problem with sparse coding (1) using proximal gradient descent (Auslender and Teboulle, 2006; Parikh and Boyd, 2014), which updates the weight vector at the $k$th iteration by:

[TABLE]

Here, $\eta>0$ is the learning rate and $G(w^{k})$ is the full gradient involving all training examples,

[TABLE]

The proximal gradient descent algorithm is capable of solving the regularized training problem (1) with $\Psi(w)=\lambda\|w\|_{1}$, while optimally fixing irrelevant weights in $w\in\mathbb{R}^{p}$ to the exact zero value. However, it will be too costly to use the method for large $n$, since it will require $n$ back-propagation steps to compute the $\nabla\ell(w^{k};x_{i},y_{i})$ for all training examples in each update.

We suggest to use a minibatch gradient $g_{B}(w)$ in the proximal operator so that the update will be:

[TABLE]

Due to the stochasticity in minibatch gradients, the behavior of the proximal gradient algorithm based on the update (2) will become less predictable, but some convergence results are available for the algorithm (Nitanda, 2014; Patrascu and Necoara, 2018; Rosasco et al., 2014) (under the assumption that the loss function $\ell$ is convex and Lipschitz continuous in $w$, the objective function value converges to the optimal value with the rate $\mathcal{O}(k^{-1/2})$ in expectation).

We further consider the update (2) within the RMSProp (Hinton, 2014) and ADAM (Kingma and Ba, 2015) algorithms, arguably the two most popular methods for training deep neural networks. RMSProp uses adaptive learning rates computed differently on gradient components, while ADAM combines the idea of adaptive learning rates and that of the momentum method (Polyak, 1964) to improve convergence and to escape saddle points. We integrate proximal operators with RMSProp and ADAM: the resulting algorithms are called Prox-RMSProp (Algorithm 1) and Prox-ADAM (Algorithm 2). We can conjecture that the search directions produced by Prox-ADAM will be more stable than those of Prox-RMSProp since the former uses compositions of minibatch gradients and momentum directions, not just noisy minibatch gradients as in the latter. We show in experiments that the behavior of Prox-ADAM is indeed more stable than that of Prox-RMSProp.

2.4 Retraining

Once we have obtained a sparse model via sparse coding, we can consider an optional step to train the weights again without any regularization, starting from the previously trained weight values, while excluding the zero-valued weights from training. This type of retraining is known as debiasing (Wright et al., 2009), which can be used to remove the unwanted reduction of weight values due to regularization. It has been shown that debiasing can improve the quality of estimation (Figueiredo et al., 2007), although it may undo some desired effect of regularization, e.g., the reduction of distortion caused by noise (Donoho, 1995). Our methods work well without retraining, but our experiments show that retraining can achieve further compression without sacrificing prediction accuracy.

3 Accelerated OpenCL Operations

In order to perform forward and backward computations of deep neural networks in an accelerated manner, we must use an efficient representation of sparse weight matrices, which can be also stored in a compact form to reduce memory footprint. In the previous research, sparse data structure and matrix operations have not been discussed fully enough (Han et al., 2015, 2016; Wen et al., 2016), due to the difficulty of using sparse matrix operations efficiently on GPUs. Therefore, implementations and testing were performed mainly on CPUs (Han et al., 2015, 2016), or some vendor-specific support was used, e.g., the cuSPARSE library, which is not an open-source software and available only for NVIDIA hardware (Wen et al., 2016).

In this section, we discuss the details how we implement sparse weight matrices and efficient sparse matrix operations for training deep neural networks with sparse coding, making use of our proposed algorithms. We base our discussion on the OpenCL-Caffe (oclcaffe, ) software, an OpenCL-capable version of the favorite open source deep learning software Caffe (Jia et al., 2014). We used OpenCL-Caffe with an open source OpenCL back-end library called ViennaCL (vcl, ), which provides efficient implementations of basic linear algebra operations and some of the sparse matrix functionality.

3.1 Compressed Sparse Matrix

In Section 2, we discussed how we could apply sparse coding to fix the values of irrelevant learning weights of DNNs at zero. However, if the zero values are explicitly stored in memory, the model will use the same amount of memory as the original model without compression. That is, for model compression, we need a special data structure to store only the nonzero weight values in GPU memory.

For our implementation, we have considered several popular formats, in particular, DIA, ELL, CSR, and COO matrix formats to store the sparse weight matrices (Bell and Garland, 2008) (see Figure 1 for comparisons of the formats). Among these, DIA is suitable for the case when nonzero values are at a small number of diagonals, and ELL is for the case when matrix rows have similar numbers of nonzero entries. Since there is usually no particular pattern of zero entries in weight matrices produced by sparse coding, we have concluded that these formats are unsuitable.

The compressed sparse row (CSR) format is probably the most popular format for representing unstructured sparse matrices. This format stores column indices and nonzero values in indices and data, respectively, while in ptr it stores the indices where new rows begin. Compared to DIA and ELL, the CSR format can store variable numbers of nonzeros in rows efficiently. The coordinate (COO) format is similar to CSR, but operations on COO can be made simpler as it also stores row indices in row for every nonzero entry. The extra storage required by COO for the row indices appears to be less economical than CSR, as our target platforms include small embedded systems. Therefore, we conclude that the CSR format will be the best for representing compressed sparse weight matrices in small, GPU-enabled devices. In ViennaCL, the CSR format is implemented as the C++ class compressed_matrix, and we have adapted this class to implement our new matrix operations for forward/backward passes of deep neural networks.

3.2 Sparse Matrix Multiplication in OpenCL

For training a deep neural network, we need efficient sparse matrix operations to deal with the compressed sparse weight matrices in the CSR format. To simplify notations, let us use $W$ to denote the sparse weight matrix that coordinates the transfer between two consecutive DNN layers, we call the bottom and the top layers (bottom $\rightarrow$ top is the forward pass direction). Denoting by $X_{B}$ and $X_{T}$ the input from the bottom layer and the output to the top layer respectively and following the shapes and orders of matrices according to the original implementation of Caffe, we can summarize the matrix multiplications needed for the forward and the backward propagation steps as follows:

[TABLE]

Here, $X_{B}$ and $\frac{\partial Loss}{\partial X_{T}}$ are typically dense matrices, and therefore we essentially need two types of operations: dense$\times$compressed*′* ($D\times C^{\prime}$ in short), where ′ is the matrix transpose operation, and dense$\times$compressed ($D\times C$ in short). Unfortunately, these operations are not available in the current version of ViennaCL (accessed on 19 October 2018): there exist $C\times D$ and $C\times D^{\prime}$ operations in ViennaCL, so we could use a workaround $(C\times D^{\prime})^{\prime}=D\times C^{\prime}$, but this requires extra memory space for transposing the result of $C\times D^{\prime}$. Furthermore, such a workaround is not available for $D\times C$ since $(D\times C)^{\prime}=C^{\prime}\times D^{\prime}$, and the transpose operation for compressed sparse matrices ($C^{\prime}$) is not available in ViennaCL. As a solution, we provide two new dense-compressed matrix multiplications accelerated by OpenCL.

3.2.1 Dense $\times$ Compressed*′*

In our implementation, this operation is used for computing forward propagation. This operation is well suited for GPU acceleration, since the compressed matrix stores nonzero elements rowwise, where we access the nonzero elements stored in the compressed matrix in a rowwise fashion to compute the inner product between a row of the dense matrix and a column of the compressed*′* matrix. Figure 2 shows a sketch of the operation and our OpenCL kernel code to perform this multiplication on GPUs: an OpenCL thread group is assigned to each row of Dmat (dense matrix), and multiple threads in the group will handle the assigned columns, one thread per column. For each (row, col) pair, an inner product between the row in Dmat and the column in Cmat*′* (compressed${}^{\prime})$ will be computed. In our discussion hereafter, we assume that Dmat and result matrices are stored in a rowwise fashion. The column memory access of Cmat*′* equals to the row access of Cmat, and therefore we can use Cmat_row_ptrs to enumerate the nonzero elements corresponding to the variable col. Each thread can access only the required nonzero elements in consecutive memory locations, and therefore thread memory access will be coalesced, leading to efficient parallelism on GPUs.

3.2.2 Dense $\times$ Compressed

This dense-compressed matrix operation is required for backward propagation. Unlike the previous one, this operation is not ideally suited for GPU parallelism since the OpenCL kernel needs to access the Cmat matrix columnwise, while the nonzero entries of the compressed matrix are stored in a rowwise fashion. Unless we store an extra array for nonzero entries stored columnwise in the compressed matrix data structure, it is unavoidable that the memory access pattern will not coalesce. Still, we can design an OpenCL kernel to parallelize the computation for each (row, col) pair, as shown in Figure 3.

3.3 Prox Operator in OpenCL

The proximal operator discussed in Section 2.2 can be performed in parallel since the outcome can be computed elementwise, as shown in Equation (2). ViennaCL has a collection of kernels for matrix unary elementwise operations, which provides a basis for implementing our own kernel. Our code in Figure 4 assigns thread groups to rows and threads in groups to columns, similar to the two OpenCL kernels we discussed above.

4 Experiments

To show the effectiveness of our compression method for deep neural networks, we tested our method on four popular convolutional neural networks on image recognition tasks: (i) Lenet-5 on the MNIST dataset (Lecun et al., 1998) and (ii) AlexNet (Krizhevsky et al., 2012), (iii) VGG16 (Simonyan and Zisserman, 2015), and (iv) ResNet-32 (He et al., 2016) on the CIFAR-10 dataset (Krizhevsky, 2009). In both datasets, we have used the original split of training and test data (MNIST has $28\times 28$ grey-scaled images with no. of train/test examples = $60k/10k$, and CIFAR-10 has $32\times 32$ color images with train/test = $50k/10k$). We fixed the number of training updates to $60k$ with a minibatch size of $128$ (so that the effective number of example iterations will be $60k\times 128$) since the number was large enough for the resulting neural networks to reached known top prediction accuracy. We compared our proposed method based on $\ell_{1}$-regularized sparse coding (denoted by SpC), to the existing pruning method (Han et al., 2015) (denoted by Pru) based on thresholding and retraining and the state-of-the-art method (Carreira-Perpiñán and Idelbayev, 2018) (denoted by MM) based on the method of multipliers optimization algorithm. To initialize weight values, we have used the initialization from He et al. (2015), which is known to work well with neural networks containing the ReLU activation.

Our implementation is available as open-source software (https://github.com/sanglee/caffe-mc-opencl).

4.1 Comparison of Training Algorithms

Our algorithms Prox-RMSProp (Algorithm 1) and Prox-ADAM (Algorithm 2) use random initialization and random minibatch examples in weight updates. Furthermore, model compression will impose statistical bias compared to the reference models without compression. Hence, it is natural to expect some variation if we repeat training with different random seeds.

Here, we compare our two algorithms in order to investigate how much variation they will exhibit in training, regarding test accuracy and compression rate (the ratio of the number of zero entries to the total number of learning parameters). We have trained Lenet-5 (MNIST), AlexNet (CIFAR-10), VGGNet (CIFAR-10), ResNet-32 (CIFAR-10) multiple times with different random seeds. Amongst the experiments, our algorithms showed the most distinctive characteristics on VGGNet, as shown in Figure 5. Overall, Prox-ADAM showed smaller variance in both test accuracy and compression rate than Prox-RMSProp. This behavior is expected since the search directions produced by Prox-ADAM is more stable than those of Prox-RMSProp: in the former, the direction is a composition of a momentum direction (which brings stability) and a minibatch gradient direction, whereas in the latter it is solely based on a noisy minibatch gradient. For this reason, we chose Prox-ADAM for the rest of our experiments.

4.2 Compression Rate and Prediction Accuracy

In the training problem with sparse coding (1), we use the $\ell_{1}$-regularizer $\Psi(w)=\lambda\|w\|_{1}$ where $\lambda>0$ is a hyperparameter that determines the amount of compression (higher $\lambda$ gives larger numbers of zero entries in $w$ and thereby higher compression rate).

Figure 6 shows test accuracy and compression rate of the four network-dataset pairs with respect to the change of the $\lambda$ parameter value, for our sparse coding approach (Figure 6a, SpC) and the existing pruning approach (Figure 6b, Pru). The test accuracy values of the reference networks (trained without sparse coding) are shown as the horizontal dotted lines. SpC: at small $\lambda$ values, we can see that the test accuracy of compressed models is usually better than that of the reference models. This may happen when the reference model is overfitting the data and $\ell_{1}$ regularization is mitigating the effect. When the test accuracy values of the compressed models were similar to those of the reference models (crossings of vertical and horizontal dotted lines), our method was able to remove about $90\%$ of the weights (one exception was ResNet-32, for which both Pru and SpC without retraining showed poor compression). The plots also indicate that further compression will be possible if we are ready to sacrifice prediction accuracy by a small margin. Pru: test accuracy values drop much more rapidly as we increase the compression rate, compared to our sparse coding approach. Similar test accuracy to the reference model has been achieved with about $40\%$ of compression in case of Lenet-5, but no compression was possible for AlexNet, VGGNet, and ResNet-32 if we wanted to achieve similar test accuracy to the reference model.

4.3 The Effect of Retraining

Compressed models can be debiased by means of an extra training the compressed networks, where the weights at the zero value are fixed and not updated during retraining. Figure 8 shows the prediction accuracy of the neural networks at different compression rates, comparing models produced by SpC and Pru, and their retrained versions, SpC(Retrain) and Pru(Retrain).

We can see that retraining is indeed a necessary step for the pruning approach (Pru) as previously known (Han et al., 2015), since otherwise compressed models show impractically low prediction performance. The pruning method with retraining, Pru(Retrain), shows similar prediction accuracy to our sparse coding method SpC (without retraining) up to a moderate compression rate. However, when the compression rate is very high, our method SpC clearly outperforms Pru(Retrain). In addition, retraining improves the prediction accuracy of our method SpC even further, especially when the compression rate is very high, which can be seen from the solid and dotted vertical lines where SpC and SpC(Retrain), respectively, achieve at least $99\%$ of the reference accuracy with maximal compression. Therefore, for our sparse coding method SpC, retraining can be used when we aim for very high compression.

Table 1 summarizes the compression and prediction results of all cases.

4.4 Comparison to the State-of-the-Art Approach

We also compare our method to the state-of-the-art network pruning approach based on the method of multipliers (Carreira-Perpiñán and Idelbayev, 2018), which we refer to as MM. In this method, one first duplicates the learning parameters $w$ with an auxiliary variable $\theta$, to convert the training regularization problem (1) into the following constrained optimization problem:

[TABLE]

where $L(w)=\frac{1}{n}\sum_{i=1}^{n}\ell(w;x_{i},y_{i})$ is the total loss and $\Psi(\theta)$ is a regularizer. Then MM constructs an augmented Lagrangian function

[TABLE]

Here, $\lambda\in\mathbb{R}^{p}$ is the Lagrange multipliers and $\mu>0$ is an auxiliary parameter. Then, MM iterates (i) the minimization of $\mathcal{L}$ (in terms of $w$ and $\theta$ in an alternating fashion) and (ii) a gradient ascent step of $\mathcal{L}$ in $\lambda$, while driving $\mu\to\infty$.

Our approach has several benefits over the MM method. First, MM requires a pre-trained model to start network pruning, whereas our method can start from random weights. Second, due to the duplication of learning weights $w\in\mathbb{R}^{p}$ by $\theta$ in Equation (3) and the introduction of the Lagrange multipliers $\lambda\in\mathbb{R}^{p}$, MM requires about double amount of memory for training than our method (ours: $(w,\nabla_{w}L(w))$ and MM: $(w,\nabla_{w}L(w),\theta,\lambda)$). Third, the convergence of MM is quite sensitive to how we control learning rates and the auxiliary parameter $\mu\to\infty$ in the augmented Lagrangian, where our method has no such sensitivity.

Table 2 shows the comparison between our method SpC and the state-of-the-art method MM, regarding two convolutional neural networks tested in the original paper (Carreira-Perpiñán and Idelbayev, 2018). Note that the comparison would be more favorable to MM because MM is allowed (i) to start from the state-of-the-art pretrained models and (ii) to use different solvers and auxiliary parameter control strategies as in the original paper (Carreira-Perpiñán and Idelbayev, 2018). Nevertheless, our method SpC shows a competent compression performance even though its compressed learning starts from random weight values.

Figure 8 shows the convergence behavior of SpC and MM. The compression is performed every update in SpC and every 4k updates in MM, and therefore the compression rate convergence may look smoother in MM. (In fact, it is another tuning parameter in MM how often it performs weight compression, and we found that the algorithm is quite sensitive to different choices.) However, our method SpC achieves the top compression rate and test accuracy much faster than MM. Therefore, our method suits better for embedded systems where we can afford only a few thousand training iterations due to time and energy constraints.

4.5 Performance on Embedded Systems

We also tested our suggested framework in an embedded system, which has a similar spec to the Samsung Galaxy S7 smartphone (Seoul, Korea). Our test system has six-core 64-bit ARM CPU, 4 GB main memory, ARM Mali-T860 GPU, OpenCL 1.2 support, and Ubuntu 16.04 operating system. In this case, we are interested in the inference time comparison between full reference models and compressed models, since in the latter the weights are stored in a compact form and will be computed with sparse matrix operations, and therefore some speedups can be expected.

Table 3 shows the results on the test embedded system, also making a comparison to the results on a workstation with an NVIDIA GPU. Despite that we have achieved some speedup, it seems to fall short considering the size of compressed models. One reason is that the compressed convolution filters have irregular nonzero patterns for which full GPU acceleration is difficult. We plan to investigate this issue further in our future research.

5 Conclusions

In this paper, we proposed an efficient model compression framework for deep neural networks based on sparse coding with $\ell_{1}$ regularization, proximal algorithms, debiasing, and compressed computation of sparse weights using OpenCL. We believe that our method will be more versatile for embedded system applications than the previous methods as it does not require a pre-trained model as an input, while it can produce competent compressed models.

\authorcontributions

Conceptualization, S.L.; formal analysis, S.L. and J.L.; software, S.L and J.L.; writing – original draft preparation, S.L.; writing–review and editing, S.L.

\funding

This work was supported by the research fund of Hanyang University (HY-2017-N).

\conflictsofinterest

The authors declare no conflict of interest.

\appendixtitles

yes

Appendix A Layer-Wise Compression Rate

Here, we check the compression rates of each layer of the three networks, Lenet-5, AlexNet, and VGGNet when we apply our sparse coding methods SpC and SpC(Retrain). We chose the regularization parameter value $\lambda$ for each method so that the compressed models achieved at least $99\%$ of the test accuracy of the reference model while maximizing compression rates among trials.

Tables 4–7 show the results. We can again confirm our discussion in the previous experiments that our method without retraining, SpC, provides superb compression rate being on par with the reference model in prediction accuracy. SpC showed compression rates: Lenet-5 ($96.9\%$), AlexNet ($90.65\%$), VGGNet ($94.41\%$), and ResNet ($20.67\%$). Further compression was also possible with retraining. SpC(Retrain) showed compression rates: Lenet-5 ($97.16\%$), AlexNet ($97.88\%$), VGGNet ($98.13\%$), and ResNet ($87.50\%$), while prediction accuracy values of compressed models were similar or slightly improved from those of SpC. From the tables, we can also observe that the layers that are near the input and the output layers are compressed less than the other layers in the middle. One could use such information to redesign and optimize the architecture of the neural networks for better computational efficiency.

\reftitle

References