Cluster Regularized Quantization for Deep Networks Compression

TL;DR

This paper introduces Cluster Regularized Quantization (CRQ), a method that compresses deep neural networks to ternary weights with minimal accuracy loss by regularizing weights during retraining.

Contribution

The paper proposes a novel cluster regularization technique that enables effective low-bit quantization of DNNs during retraining, reducing quantization error.

Findings

Effective ternary quantization with minimal accuracy drop

Reduces model size and computational cost

Demonstrated on benchmark datasets

Abstract

Deep neural networks (DNNs) have achieved great success in a wide range of computer vision areas, but the applications to mobile devices is limited due to their high storage and computational cost. Much efforts have been devoted to compress DNNs. In this paper, we propose a simple yet effective method for deep networks compression, named Cluster Regularized Quantization (CRQ), which can reduce the presentation precision of a full-precision model to ternary values without significant accuracy drop. In particular, the proposed method aims at reducing the quantization error by introducing a cluster regularization term, which is imposed on the full-precision weights to enable them naturally concentrate around the target values. Through explicitly regularizing the weights during the re-training stage, the full-precision model can achieve the smooth transition to the low-bit one. Comprehensive experiments on benchmark datasets demonstrate the effectiveness of the proposed method.

Tables

Table 1. Table 1 : Top-1 and Top-5 error rate (%) of ternary AlexNet on ImageNet.

Method	Top-1	Top-1 $\downarrow$	Top-5	Top-5 $\downarrow$
Ref [16]	42.80	-	19.70	-
TWN [12]	45.50	-2.70	23.20	-3.50
TTQ [16]	42.50	0.30	20.30	-0.60
Our Ref	42.77	-	19.79	-
Our CRQ	42.02	0.75	19.18	0.61

Table 2. Table 2 : Top-1 and Top-5 error rate (%) of ternary ResNet-18 on ImageNet.

Method	Top-1	Top-1 $\downarrow$	Top-5	Top-5 $\downarrow$
Ref [16]	30.40	-	10.80	-
TWN [12]	34.70	-4.30	13.80	-3.00
TTQ [16]	33.40	-3.00	12.80	-2.00
Our Ref	30.89	-	11.18	-
Our CRQ	32.88	-1.99	12.56	-1.38

Table 3. Table 3 : Error rate (%) of ternary ResNet-32 and ResNet-44 on CIFAR-10.

Model	Method	Ref	Err.	Err. $\downarrow$
resnet-32	TTQ [16]	7.67	7.63	0.04
resnet-32	CRQ	7.74	7.61	0.13
resnet-44	TTQ	7.18	7.02	0.16
resnet-44	CRQ	7.21	6.95	0.26

Equations

$$\underset{W_{\mathcal{Q}}}{\min }\frac{1}{2N}||W-W_{\mathcal{Q}}|{|}_2^2$$

$$\underset{W_{\mathcal{Q}}}{\min }\frac{1}{2N}||XW-X{W}_{\mathcal{Q}}|{|}_2^2$$

$$\begin{array}{c}J(Z,\alpha )=\Vert W-\alpha M^{T}Z{\Vert }^2\\ {\alpha }^{\ast },Z^{\ast }=\underset{\alpha ,Z}{\mathrm{argmin}}J(Z,\alpha )\\ s.t.Z\in \{0,1{\}}^{3\times N},\sum _i^3{Z}_{i,j}=1,\forall j\in [1,N]\end{array}$$

$$\begin{array}{c}\underset{Z}{\min }Tr(W^{T}W)-2\alpha Tr(W^T{M}^{T}Z)+{\alpha }^{2}Tr(Z^{T}M{M}^{T}Z)\\ s.t.Z\in \{0,1{\}}^{3\times N},\sum _i^3{Z}_{i,j}=1,\forall j\in [1,N]\end{array}$$

$$Tr(W^T{M}^{T}Z)$$

$$=C+z^{T}Mw$$

$$Tr(Z^{T}M{M}^{T}Z)$$

$$=C+M^{T}z{z}^{T}M$$

$$\begin{array}{c}\underset{q}{\min }{\alpha }^2{q}^2-2\alpha wq+C\\ s.t.z\in \{0,1{\}}^{3\times 1},\sum _i^3{z}_i=1\end{array}$$

$$z_i=\{\begin{array}{rl}1& i=H(M,\frac{w}{\alpha })\\ 0& otherwise\end{array}$$

$$\alpha =\frac{W{Z}^{T}M}{M^{T}Z{Z}^{T}M}$$

$$\begin{array}{c}\underset{W,Z,\alpha }{\min }L(W)+\lambda J(Z,\alpha )\\ s.t.Z\in \{0,1{\}}^{3\times N},\sum _i^3{Z}_{i,j}=1,\forall j\in [1,N]\end{array}$$

$$W^t=W^{t-1}-\eta (\frac{\partial L}{\partial W^{t-1}}+\lambda (W^{t-1}-\alpha M^{T}Z))$$

Full text

**Index Terms— ** deep neural networks, object classification, model compression, quantization

1 Introduction

In the past few years, DNNs have been widely applied to various computer vision tasks, e.g., image classification, object detection, action recognition. However, it is very difficult to deploy these deep models on resource-constrained mobile devices due to their high storage and computational cost.

In order to make DNNs available on resource-constrained devices, many model compression and acceleration methods have been proposed including weight pruning [1, 2], low-rank approximation [3, 4], and efficient model designs [5, 6]. Another line of works [7, 8] for network compression is to reduce the presentation precision of the full-precision parameters. Going one step further, restricting weights to some values with special types (e.g., zero or powers of two) can achieve fast inference on embedded devices such as the Field Programmable Gate Array (FPGA). Existing methods [9, 10] usually quantize weights by directly approximating the full-precision weights with the low-bit ones. XNOR-Net [11] and TWN [12] minimize the reconstruction error of the outputs between the full-precision model and the quantized one. These methods suffer from a common limitation: the reconstruction error of weights or intermediate outputs do not always truly reflect the change of the classification loss. Thus, directly cutting or sharing weights according to the reconstruction error of a layer may cause serious quantization error, which would bring significant disturbance to the classification loss and hurt the prediction accuracy of the quantized model.

To address the problem, we introduce an cluster regularization term, through which the full-precision weights can naturally concentrate around the target values during the re-training stage. After this, weights of each layer are quantized into corresponding cluster centers. Consequently, the quantization error can be significantly reduced.

Our main contributions are summarized as follows. First, we present a new quantization method for deep networks compression by the introduction of the cluster regularization. Second, comprehensive experiments on benchmark datasets show that the proposed method can achieve competitive performance in accuracy and the rate of convergence.

2 Related work

The goal of network quantization is to replace the full-resolution weights with low-bit ones without significant accuracy drop. This line of works can be traced back to two representative methods: Expectation Back-Propagation (EBP) [13] and BinaryConnect [10] in which authors explicitly constrained the weights to either $+1$ or $-1$ during propagations. These two methods achieve decent performance on small datasets including MNIST and CIFAR-10, yet they suffer from significant accuracy drop on challenging datasets such as ILSVRC-12 [14]. Afterwards, plenty of works were proposed to improve the performance of BinaryConnect. Rastegari et al. presented XNOR-Network [11] that approximated the full-precision weights by introducing a scaling factor during binarization. For pursuing higher accuracy, High-Order Residual Quantization (HORQ) [15] sought to compensate the information loss of binary quantization by conducting convolutional operations on inputs in different scales and then combined the results. The Ternary Weight Network (TWN) [12] introduced zero as a third quantized value and was the first method that achieved decent results on the ILSVRC-12 dataset. Trained Ternary Quantization (TTQ) [16] achieved the state-of-the-art performance by taking the scaling factors as trainable parameters. Different from these works, our approach considers further reducing the quantization error to bridge the accuracy gap by introducing the cluster regularization with little computational complexity.

3 Our Method

For better description of the proposed method, some notations are given first. Considering a layer of the DNN model, $X$ represents the input tensor, $W$ denotes the weight of the layer and $W_{\mathcal{Q}}$ represents the quantized weights of the layer.

3.1 Motivation

Most existing low-bit quantization methods [9, 10, 11, 12] can be classified into two categories. The first category such as BinaryNet [9] and BinaryConnect [10] directly approximates the full-precision weights with the low-bit ones:

[TABLE]

where $N$ is the size of $W$. The second category such as XNOR-Net [11] and TWN [12] usually minimizes the reconstruction error of a layer, which can be defined as the Euclidean distance of the outputs between the full-precision model and the quantized one:

[TABLE]

These methods suffer from a common limitation: the reconstruction error of weights or outputs between the full-precision model and the quantized one don’t always truly reflect the change of the classification loss. Thus, directly cutting or sharing weights via the reconstruction error of a layer may cause serious quantization error, which would bring significant disturbance to the classification loss and hurt the prediction accuracy of the quantized model. This inspires us to define an additional regularization term, which is imposed on the weights of the full-precision model to enable them naturally concentrate around the target values by re-training. Particularly, the additional regularization can assist to reduce the quantization error and achieve the smooth transition from the full-precision model to the low-bit one.

3.2 Formulation

To softly modify the weights distribution by re-training, an cluster regularization term is defined. Han et al. [17] apply $k$-means to assign weights with $K$ clusters layer-wise at first. Then the weights belonging to the same cluster centers share a same value. Unlike them, the values of cluster centers in our method are restricted to $T=\{-\alpha,0,+\alpha\}$, where $\alpha\in\mathbb{R}$ is a scaling factor. Without loss of generality, assuming $W$ is a row vector in $\mathbb{R}^{1\times N}$ and $M=[-1,0,1]^{T}$ is a column vector, where $N=c\times h\times z$, $c$, $h$ and $z$ represents the number of channels, the height and the width respectively. To partition weights $W$ into three clusters whose cluster centers are chosen from $T$, we solve the following optimization problem:

[TABLE]

where $Z\in\{0,1\}^{3\times N}$ represents the cluster indicator, and the summation of each column is equal to $1$. It’s worth noting that the optimization problem is different from the traditional $k$-means, since the cluster centers are restricted to the discrete values chosen from $T$.

3.3 Optimization Algorithm

Eq. 3 is non-convex, and optimization parameters involve $\alpha$ and $Z$. We consider using the alternating algorithm to solve this problem, i.e., solve $\alpha$ when $Z$ is fixed and vice versa.

Solve $Z$ with $\alpha$ fixed. By fixing $\alpha$, Eq. 3 is written as:

[TABLE]

Directly solving $Z$ for Eq. 4 is difficult due to its discrete constraints. Inspired by [18], $Z$ can be optimized column-wise by fixing other columns. In this way, $Z$ can be solved in roughly square time by learning only one column each time. Solving Eq. 4 is equivalent to optimizing two following general terms, i.e., $Tr(W^{T}M^{T}Z)$ and $Tr(Z^{T}MM^{T}Z)$. Let $w$ be the $i^{th}$ column of $W$ and $\tilde{W}$ indicates the vector of $W$ excluding $w$, $i=1,2,\cdots,N$. Similarly, $z$ denotes the $i^{th}$ column of $Z$ and $\tilde{Z}$ represents the matrix of $Z$ excluding $z$. Then the following transformation can be got:

[TABLE]

Here, those terms that don’t contain $z$ are set to a constant $C$. Similarly, we have:

[TABLE]

According to Eq. 5 and Eq. 6, Eq. 4 can be written as follows:

[TABLE]

where $q=M^{T}z$ and $q,\alpha,w\in\mathbb{R}$. Assuming Eq. 7 is convex, it has the optimal solution when $q$ is equal to $\frac{w}{\alpha}$. But for this problem, $q$ is discrete. Let $H(M,\frac{w}{\alpha})$ returns the index of the nearest value of $M$ to $\frac{w}{\alpha}$, the optimal $z$ is solved as:

[TABLE]

Each column of $Z$ is optimized according to Eq. 8. Then $Z$ can be solved after $N$ iterations.

Solve $\alpha$ with $Z$ fixed. Eq. 4 is convex when $Z$ is fixed. Eq. 4 has the optimal solution when its derivative w.r.t. $\alpha$ is equal to zero. Thus we have:

[TABLE]

In this way, $\alpha$ and $Z$ are updated iteratively until convergence. Here, it’s not hard to find that solving $\alpha$ via Eq. 9 is equivalent to computing $\alpha$ as $\frac{1}{|I|}\sum_{i\in I}|W_{i}|$, where $I$ is the index sets of weights belonging to the cluster centers $-\alpha$ and $+\alpha$. Thus, solving Eq. 3 using the alternating algorithm brings little computational complexity.

As mentioned above, Eq. 3 is used as a regularization term imposed on $W$ during the re-training stage. Assuming $L(W)$ is the loss function of the full-precision model, the objective function of the re-training stage is defined as follows:

[TABLE]

where $\lambda$ is a positive coefficient balancing the regularization term. Eq. 10 can be solved by iteratively updating $W$, $Z$ and $\alpha$. First, $Z$ and $\alpha$ are solved according to Eq. 4 - 9. Then $W$ is updated by stochastic gradient descent (SGD). We can derive the following weight update method:

[TABLE]

where $\eta$ is the learning rate. By the interaction among $Z$, $\alpha$ and $W$ during the re-training stage, the proposed regularization term enables $W$ naturally concentrate around the target values chosen from $T$.

3.4 The Whole Quantization Framework

After retraining, weights of each layer are quantized into corresponding cluster centers according to the indicator matrix $Z$ (Eq. 3), for parameter sharing. Specifically, the quantized weights are exactly equal to $\alpha M^{T}Z$ when Eq.3 has the optimal solution. In contrast, the re-training strategy with the cluster regularization brings less quantization error than the reconstruction-based methods [11, 15], since the weight $W$ remains in a highly clustered state after re-training. To further reduce the effects of quantization error to the classification loss, we fine-tune the re-trained model for several epochs. As in [10, 12], the quantized weights is used in the forward and backward process, while the full-precision weights are used in the stage of parameters update. The whole quantization framework is shown in Algorithm 1.

The pipeline of [19, 20] is close to ours, but we have to emphasize that our approach has three critical differences with them. First, we define and solve a new cluster optimization problem, in which the cluster centers are restricted to several discrete values. Solving the problem is very meaningful to the low-bit quantization, while the optimization algorithms in [19, 20] are not applicable to the special problem any more. Second, we seek to solve the cluster indicator $Z$ and the scaling factor $\alpha$ by the alternating algorithm, which avoids extremely time-consuming singular factorization operations. Third, our approach introduces fewer hyper-parameters and is easy to converge.

4 Experiments

In this section, we conduct comparison experiments on CIFAR-10 [21] and ImageNet [14]. The proposed CRQ is compared with two state-of-the-art methods including TWN [12] and TTQ [16]. Our approach is implemented using the PyTorch [22] framework. For the experiments on CIFAR-10 and ImageNet, we adopt the same settings as in [16]. Besides, $\lambda$ (Eq. 10) is set to $0.001$.

4.1 ImageNet

Table 1 reports the comparison results of different methods on AlexNet. It can be observed that our ternary AlexNet can reach Top-1 error rate of $42.02\%$ and Top-5 error rate of $19.18\%$, which achieves the best performance compared with other methods and even outperforms its reference model by $0.75\%$ in Top-1 error rate. Next, further comparative experiments are conducted on ResNet models. Table 2 summarizes the comparison results of different methods on ResNet-18. It’s worth noting that our approach can still get a ternary ResNet-18 that has minor increase of the error rate: $1.99\%$ of Top-1 error rate and $1.38\%$ of Top-5 error rate, which are lower than TWN and TTQ.

4.2 CIFAR-10

Table 3 shows the comparison results with TTQ on ResNet-32 and ResNet-44. Similar to the results on ImageNet, our method achieves bigger error rate drop than TTQ for the ternary ResNet-32 and ResNet-44. Particularly, our ternary ResNet-44 even outperforms the full-precision model by $0.26\%$ in the error rate, which may be resulted from the ability of the cluster regularization. Above experiments also imply that our CRQ can deal with the problem of ternary quantization with increased network depth.

4.3 Convergence Analysis

Going one step further, the rate of convergence of different methods is compared. We compare the evolution process of the training accuracy of different methods. Figure 1 shows the training accuracy of ternary ResNet-20 changes over epochs on CIFAR-10. We can find that TWN, TTQ and our approach can converge to a minor error, but their accuracy curves are not as smooth as ours. Besides, our method converges within fewer epochs. These results demonstrate our approach has a faster rate of convergence.

5 Conclusion

This paper proposes Cluster Regularized Quantization (CRQ), a ternary quantization method for deep networks compression. The key contribution is to define a new cluster regularization term, which significantly reduces the quantization error. Comprehensive experiments on CIFAR-10 and ImageNet demonstrate the CRQ can achieve competitive performance in accuracy and the rate of convergence.

6 Acknowledgment

This work has been supported by the National Key Research and Development Program of China No. 2018YFD0400902 and the National Natural Science Foundation of China (Grant No. 61673376 and 61573349).