Data-Free Quantization Through Weight Equalization and Bias Correction

TL;DR

This paper presents a data-free quantization method for deep neural networks that maintains near-original performance without fine-tuning, using weight equalization and bias correction, applicable to various architectures and tasks.

Contribution

The authors introduce a novel data-free quantization technique leveraging weight equalization and bias correction, eliminating the need for fine-tuning or hyperparameter tuning.

Findings

Achieves near-original model performance on common architectures.

Extends effectively to tasks like semantic segmentation and object detection.

State-of-the-art results on MobileNet family models.

Abstract

We introduce a data-free quantization method for deep neural networks that does not require fine-tuning or hyperparameter selection. It achieves near-original model performance on common computer vision architectures and tasks. 8-bit fixed-point quantization is essential for efficient inference on modern deep learning hardware. However, quantizing models to run in 8-bit is a non-trivial task, frequently leading to either significant performance reduction or engineering time spent on training a network to be amenable to quantization. Our approach relies on equalizing the weight ranges in the network by making use of a scale-equivariance property of activation functions. In addition the method corrects biases in the error that are introduced during quantization. This improves quantization accuracy performance, and can be applied to many common computer vision architectures with a straight forward API call. For common architectures, such as the MobileNet family, we achieve state-of-the-art quantized model performance. We further show that the method also extends to other computer vision architectures and tasks such as semantic segmentation and object detection.

Tables

Table 1. Table 1: Top1 ImageNet validation results for MobileNetV2, evaluated at full precision and 8-bit integer quantized. Per-channel quantization is our own implementation of [ 16 ] applied post-training.

Model	FP32	INT8
Original model	71.72%	0.12%
Replace ReLU6	71.70%	0.11%
+ equalization	71.70%	69.91%
+ absorbing bias	71.57%	70.92%
Per channel quantization	71.72%	70.65%

Table 2. Table 2: Top1 ImageNet validation results for MobileNetV2, evaluated at full precision and 8-bit integer quantized. Bold results show the best result for each column in each cell.

Model	FP32	INT8
Original Model	71.72%	0.12%
Bias Corr	71.72%	52.02%
Clip @ 15	67.06%	2.55%
+ Bias Corr	71.15%	70.43%
Rescaling + Bias Absorption	71.57%	70.92%
+ Bias Corr	71.57%	71.19%

Table 3. Table 3: DeeplabV3+ (MobileNetV2 backend) on Pascal VOC segmentation challenge. Mean intersection over union (mIOU) evaluated at full precision and 8-bit integer quantized. Per-channel quantization is our own implementation of [ 16 ] applied post-training.

Model	FP32	INT8
Original model	72.94	41.40
DFQ (ours)	72.45	72.33
Per-channel quantization	72.94	71.44

Table 4. Table 4: MobileNetV2 SSD-lite on Pascal VOC object detection challange. Mean average precision (mAP) evaluated at full precision and 8-bit integer quantized. Per-channel quantization is our own implementation of [ 16 ] applied post-training.

Model	FP32	INT8
Original model	68.47	10.63
DFQ (ours)	68.56	67.91
Per-channel quantization	68.47	67.52

Table 5. Table 5: Top1 ImageNet validation results for different models and quantization approaches. The top half compares level 1 approaches ( ∼ similar-to \sim D: data free, ∼ similar-to \sim BP: backpropagation-free, ∼ similar-to \sim AC: Architecture change free) whereas in the second half we also compare to higher level approaches in literature. Results with ∗ indicates our own implementation since results are not provided, ^results provided by [ 18 ] and † results from table 2 in [ 21 ] .

	$\sim$D	$\sim$BP	$\sim$AC	MobileNetV2		MobileNetV1		ResNet18
				FP32	INT8	FP32	INT8	FP32	INT8	INT6
DFQ (ours)	✓	✓	✓	71.7%	71.2%	70.8%	70.5%	69.7%	69.7%	66.3%
Per-layer [18]	✓	✓	✓	71.9%	0.1%	70.9%	0.1%	69.7%	69.2%∗	63.8%∗
Per-channel [18]	✓	✓	✓	71.9%	69.7%	70.9%	70.3%	69.7%	69.6%∗	67.5%∗
QT [16] ^	✗	✗	✓	71.9%	70.9%	70.9%	70.0%	-	70.3%†	67.3%†
SR+DR†	✗	✗	✓	-	-	-	71.3%	-	68.2%	59.3%
QMN [31]	✗	✗	✗	-	-	70.8%	68.0%	-	-	-
RQ [21]	✗	✗	✗	-	-	-	70.4%	-	69.9%	68.6%

Table 6. Table 6: Top1 ImageNet validation results for MobileNetV2 with weights and activations quantized to INT8. Comparing analytic and empirical bias correction combined with cross-layer equalization (CLE), bias absorption (BA) and clipping.

Model	CLE+BA	Clip@15
No BiasCorr	70.92%	2.55%
Analytic BiasCorr	71.19%	70.43%
Empirical BiasCorr	71.15%	69.85%

Table 7. Table 7: Top1 ImageNet validation results for MobileNetV2 after applying DFQ. Weights and activations quantized using symmetric and asymmetric 8-bit integer quantization.

Model	Symmetric	Asymmetric
MobileNet V1	70.32%	70.51%
MobileNet V2	71.15%	71.19%
Resnet18	69.50%	69.62%

Table 8. Table 8: Top1 ImageNet validation results for MobileNetV2 using 8-bit per-channel quantization. Activations are quantized per tensor. Showing the effect of cross-layer equalization (CLE) and bias absorption (BA) in combination with bias correction.

Model	No BiasCorr	BiasCorr
Original model	70.65%	70.80%
CLE	70.93%	71.30%
CLE+BA	71.03%	71.33%

Equations

$$\mathbb{E}[{\tilde{\mathbf{y}}}_j-{\mathbf{y}}_j]$$

$$f(sx)=sf(x)$$

$$\text{ReLU}(x)=\{\begin{array}{ll}x& \text{if }x>0\\ 0& \text{if }x\le 0.\end{array}$$

$$f(x)=\{\begin{array}{ll}{a}_{1}x+b_1& \text{if }x\le c_1\\ a_{2}x+b_2& \text{if }{c}_1<x\le c_2\\ & ⋮\\ a_{n}x+b_n& \text{if }{c}_{n-1}<x\end{array}$$

$$\mathbf{y}$$

$$=f({\mathbf{W}}^{(2)}\mathbf{S}\hat{f}({\mathbf{S}}^{-1}{\mathbf{W}}^{(1)}\mathbf{x}+{\mathbf{S}}^{-1}{\mathbf{b}}^{(1)})+{\mathbf{b}}^{(2)})$$

$$=f({\hat{\mathbf{W}}}^{(2)}\hat{f}({\hat{\mathbf{W}}}^{(1)}\mathbf{x}+{\hat{\mathbf{b}}}^{(1)})+{\mathbf{b}}^{(2)})$$

$${\hat{\mathbf{p}}}_i^{(1)}=\frac{{\hat{\mathbf{r}}}_i^{(1)}}{{\hat{R}}^{(1)}}$$

$$\underset{\mathbf{S}}{\max }\sum _i{\hat{\mathbf{p}}}_i^{(1)}{\hat{\mathbf{p}}}_i^{(2)}$$

$$\underset{j}{\mathrm{argmax}}\frac{1}{{\mathbf{s}}_j}{\mathbf{r}}_j^{(1)}=\underset{k}{\mathrm{argmax}}{\mathbf{s}}_k{\mathbf{r}}_k^{(2)}$$

$${\mathbf{s}}_i=\frac{1}{{\mathbf{r}}_i^{(2)}}\sqrt{{\mathbf{r}}_i^{(1)}{\mathbf{r}}_i^{(2)}}$$

$$\mathbf{y}$$

$$={\mathbf{W}}^{(2)}(r({\mathbf{W}}^{(1)}\mathbf{x}+{\mathbf{b}}^{(1)})+\mathbf{c}-\mathbf{c})+{\mathbf{b}}^{(2)}$$

$$={\mathbf{W}}^{(2)}(r({\mathbf{W}}^{(1)}\mathbf{x}+{\hat{\mathbf{b}}}^{(1)})+\mathbf{c})+{\mathbf{b}}^{(2)}$$

$$={\mathbf{W}}^{(2)}\hat{\mathbf{h}}+{\hat{\mathbf{b}}}^{(2)}$$

$$\mathbb{E}[\mathbf{y}]$$

$$=\mathbb{E}[\tilde{\mathbf{y}}]-\mathbb{E}[\mathbf{ϵ}\mathbf{x}].$$

$$\mathbb{E}[{\mathbf{x}}_c]$$

$$={\mathbf{\gamma }}_c\mathcal{N}\left(\frac{-{\mathbf{\beta }}_c}{{\mathbf{\gamma }}_c}\right)+{\mathbf{\beta }}_c\left[1-\Phi \left(\frac{-{\mathbf{\beta }}_c}{{\mathbf{\gamma }}_c}\right)\right]$$

$${\hat{\mathbf{r}}}^{(1)}$$

$${\hat{\mathbf{r}}}^{(2)}$$

$${\hat{R}}^{(k)}$$

$$\underset{\mathbf{S}}{\max }\sum _i{\hat{\mathbf{p}}}_i^{(1)}{\hat{\mathbf{p}}}_i^{(2)}=\underset{\mathbf{S}}{\max }\sum _i\frac{{\hat{\mathbf{r}}}_i^{(1)}{\hat{\mathbf{r}}}_i^{(2)}}{{\hat{R}}^{(1)}{\hat{R}}^{(2)}}$$

$$=\underset{\mathbf{S}}{\max }\sum _i\frac{\frac{1}{{\mathbf{s}}_i}{\mathbf{r}}_i^{(1)}\cdot {\mathbf{s}}_i{\mathbf{r}}_i^{(2)}}{{\max }_j(\frac{1}{{\mathbf{s}}_j}{\mathbf{r}}_j^{(1)})\cdot {\max }_k({\mathbf{s}}_k{\mathbf{r}}_k^{(2)})}$$

$$=\sum _i{\mathbf{r}}_i^{(1)}{\mathbf{r}}_i^{(2)}\underset{\mathbf{S}}{\max }\frac{1}{{\max }_j(\frac{1}{{\mathbf{s}}_j}{\mathbf{r}}_j^{(1)})\cdot {\max }_k({\mathbf{s}}_k{\mathbf{r}}_k^{(2)})}.$$

$$\underset{\mathbf{S}}{\min }\left(\underset{j}{\max }\left(\frac{1}{{\mathbf{s}}_j}{\mathbf{r}}_j^{(1)}\right)\cdot \underset{k}{\max }({\mathbf{s}}_k{\mathbf{r}}_k^{(2)})\right)$$

$$\underset{j}{\mathrm{argmax}}\frac{1}{{\mathbf{s}}_j}{\mathbf{r}}_j^{(1)}=\underset{k}{\mathrm{argmax}}{\mathbf{s}}_k{\mathbf{r}}_k^{(2)}$$

$${\mathbf{s}}_i=\frac{1}{{\mathbf{r}}_i^{(2)}}\sqrt{{\mathbf{r}}_i^{(1)}{\mathbf{r}}_i^{(2)}}$$

$${\left[\mathbf{ϵ}\ast \mathbb{E}[\mathbf{x}]\right]}_{c_{o}ij}$$

$$=\sum _{c_i}\left[\mathbb{E}[{\mathbf{x}}_{c_i}]\sum _{mn}{\mathbf{ϵ}}_{c_o{c}_{i}mn}\right]$$

$${\mu }_{ab}^c$$

$${{\sigma }_{ab}^c}^2$$

$${\mu }_{ab}^c$$

$$=a{\int }_{-\infty }^{a}p(x)dx+{\int }_a^{b}xp(x)dx+b{\int }_b^{\infty }p(x)dx$$

$${\int }_a^{b}xp(x)dx$$

$$={-C{\sigma }^2{e}^{-\frac{1}{2{\sigma }^2}(x-\mu {)}^2}|}_a^b+\mu (\Phi (\beta )-\Phi (\alpha ))$$

$$=\sigma \left(\varphi (\alpha )-\varphi (\beta )\right)+\mu (\Phi (\beta )-\Phi (\alpha ))$$

$${\mu }_{ab}^c=$$

Full text

1 Introduction

In recent years, deep learning based computer vision models have moved from research labs into the cloud and onto edge devices. As a result, power consumption and latency of deep learning inference have become an important concern. For this reason fixed-point quantization is often employed to make inference more efficient. By quantizing floating point values onto a regularly spaced grid, the original floating point values can be approximated by a set of integers, a scaling factor, and an optional zero point offset [16]. This allows for the use of faster and more power-efficient integer operations in matrix multiplication and convolution computations, at the expense of lower representational power. We refer the reader to [18] for details on commonly used, hardware-friendly quantization methods for deep learning models.

Quantization of 32-bit full precision (FP32) models into 8-bit fixed point (INT8) introduces quantization noise on the weights and activations, which often leads to reduced model performance. This performance degradation ranges from very minor to catastrophic. To minimize the quantization noise, a wide range of different methods have been introduced in the literature (see Section 2). A major drawback of these quantization methods is their reliance on data and fine-tuning. As an example, consider real-world actors that manage hardware for quantized models, such as cloud-based deep learning inference providers or cellphone manufacturers. To provide a general use quantization service they would have to receive data from the customers to fine-tune the models, or rely on their customers to do the quantization. In either case, this can add a difficult step to the process. For such stakeholders it would be preferable if FP32 models could be converted directly to INT8, without needing the know-how, data or compute necessary for running traditional quantization methods. Even for model developers that have the capability to quantize their own models, automation would save significant time.

In this paper, we introduce a quantization approach that does not require data, fine-tuning or hyperparameter tuning, resulting in accuracy improvement with a simple API call. Despite these restrictions we achieve near-original model performance when quantizing FP32 models to INT8. This is achieved by adapting the weight tensors of pre-trained models such that they are more amenable to quantization, and by correcting for the bias of the error that is introduced when quantizing models. We show significant improvements in quantization performance on a wide range of computer vision models previously thought to be difficult to quantize without fine-tuning.

Levels of quantization solutions

In literature the practical application of proposed quantization methods is rarely discussed. To distinguish between the differences in applicability of quantization methods, we introduce four levels of quantization solutions, in decreasing order of practical applicability. Our hope is that this will enable other authors to explore solutions for each level, and makes the comparison between methods more fair. The axes for comparison are whether or not a method requires data, whether or not a method requires error backpropagation on the quantized model, and whether or not a method is generally applicable for any architecture or requires significant model re-working. We use the following definitions throughout the paper:

Level 1

No data and no backpropagation required. Method works for any model. As simple as an API call that only looks at the model definition and weights.

Level 2

Requires data but no backpropagation. Works for any model. The data is used e.g. to re-calibrate batch normalization statistics [27] or to compute layer-wise loss functions to improve quantization performance. However, no fine-tuning pipeline is required.

Level 3

Requires data and backpropagation. Works for any model. Models can be quantized but need fine-tuning to reach acceptable performance. Often requires hyperparameter tuning for optimal performance. These methods require a full training pipeline (e.g. [16, 35]).

Level 4

Requires data and backpropagation. Only works for specific models. In this case, the network architecture needs non-trivial reworking, and/or the architecture needs to be trained from scratch with quantization in mind (e.g. [4, 31, 21]). Takes significant extra training-time and hyperparameter tuning to work.

2 Background and related work

There are several works that describe quantization and improving networks for lower bit inference and deployment [9, 10, 16, 35]. These methods all rely on fine-tuning, making them level 3 methods, whereas data-free quantization improves performance similarly without that requirement. Our method is complementary to these and can be applied as a pre-processing before quantization aware fine-tuning.

In a whitepaper, Krishnamoorthi [18], introduces a level 1 ‘per-channel’ quantization scheme, in which the weights of a convolutional weight tensor are quantized per output channel. A major drawback of this method is that it is not supported on all hardware, and that it creates unnecessary overhead in the computation due to the necessity of scale and offset values for each channel individually. We show that our method improves on per-channel quantization, while keeping a single set of scale and offset values for the whole weight tensor instead.

Other methods to improve quantization need architecture changes or training with quantization in mind from the start [1, 21, 31, 33, 36]. These methods are even more involved than doing quantization and fine-tuning. They also incur a relatively large overhead during training because of sampling and noisy optimization, and introduce extra hyperparameters to optimize. This makes them level 4 methods.

Methods that binarize [5, 15, 27, 28] or ternarize [19] networks result in models with great inference efficiency as expensive multiplications and additions are replaced by bit-shift operations. However, quantizing models to binary often leads to strong performance degradation. Generally they need to be trained from scratch, making them level 4 methods.

Other approaches use low-bit floating point operations instead of integer operations, or other custom quantization implementations [8, 17, 24, 35]. We do not consider such approaches as the hardware implementation is less efficient.

In concurrent work, Meller et al. [22] also exploits the scale equivariance of the ReLU function to rescale weight channels and notice the biased error introduced by weight quantization [7], leading to a method that resembles our data-free quantization approach. Stock et al. [32] also use the scale equivariance property of the ReLU function, but use it for network optimization instead.

3 Motivation

While many trained FP32 models can be quantized to INT8 without much loss in performance, some models exhibit a significant drop in performance after quantization ([18, 31]). For example, when quantizing a trained MobileNetV2 [30] model, Krishnamoorthi [18] reports a drop in top-1 accuracy from 70.9% to 0.1% on the ImageNet [29] validation set. The author restores near original model performance by either applying per-channel quantization, fine-tuning or both.

3.1 Weight tensor channel ranges

The fact that per-channel quantization yields much better performance on MobileNetV2 than per-tensor quantization suggests that, in some layers, the weight distributions differ so strongly between output channels that the same set of quantization parameters cannot be used to quantize the full weight tensor effectively. For example, in the case where one channel has weights in the range $[-128,128]$ and another channel has weights in the range $(-0.5,0.5)$, the weights in the latter channel will all be quantized to [math] when quantizing to 8-bits.

Figure 2 shows that large differences in output channel weight ranges do indeed occur in a (trained) MobileNetV2 model. This figure shows the weight distribution of the output channel weights of the depthwise-separable layer in the model’s first inverted residual block. Due to the strong differences between channel weight ranges that this layer exhibits, it cannot be quantized with reasonable accuracy for each channel. Several layers in the network suffer from this problem, making the overall model difficult to quantize.

We conjecture that performance of trained models after quantization can be improved by adjusting the weights for each output channel such that their ranges are more similar. We provide a level 1 method to achieve this without changing the FP32 model output in section 4.1.

3.2 Biased quantization error

A common assumption in literature (e.g. [2]) is that quantization error is unbiased and thus cancels out in a layer’s output, ensuring that the mean of a layer’s output does not change as a result of quantization. However, as we will show in this section, the quantization error on the weights might introduce biased error on the corresponding outputs. This shifts the input distribution of the next layer, which may cause unpredictable effects.

The biased error in a quantized layer’s output unit $j$ can be computed empirically using $N$ input data points as:

[TABLE]

where $\mathbf{y}_{j}$ and $\mathbf{\widetilde{y}}_{j}$ are the original outputs and the outputs generated using the quantized weight matrix, respectively.

Figure 3 shows the biased error per channel of a depthwise-separable convolution layer in a trained MobileNetV2 model. From this plot it is clear that for many channels in the layer’s output, the error introduced by weight quantization is biased, and influences the output statistics. Depthwise-separable layers are especially susceptible to this biased error effect as each output channel has only 9 corresponding weights.

Such a biased error on the outputs can be introduced in many settings, e.g. when weights or activations are clipped [23], or in non-quantization approaches, such as weight tensor factorization or channel pruning [13, 34].

In section 4.2 we introduce a method to correct for this bias. Furthermore, we show that a model’s batch normalization parameters can be used to compute the expected biased error on the output, yielding a level 1 method to fix the biased error introduced by quantization.

4 Method

Our proposed data-free quantization method (DFQ) consists of three steps, on top of the normal quantization. The overall flow of the algorithm is shown in Figure 4.

4.1 Cross-layer range equalization

Positive scaling equivariance

We observe that for a ReLU [25] activation function $f(\cdot)$ the following scaling equivariance property holds:

[TABLE]

for any non-negative real number $s$. This follows from the definition of the ReLU:

[TABLE]

This equivariance also holds for the PreLU [11] activation function. More generally, the positive scaling equivariance can be relaxed to $f(sx)=s\hat{f}(x)$ for any piece-wise linear activation functions:

[TABLE]

where $\hat{f}(\cdot)$ is parameterized as $\hat{a}_{i}=a_{i}$, $\hat{b}_{i}=b_{i}/s$ and $\hat{c}_{i}=c_{i}/s$. Note that contrary to equivariance defined in eq. 2 we now also change the function $f(\cdot)$ into $\hat{f}(\cdot)$.

4.1.1 Scaling equivariance in neural networks

The positive scaling equivariance can be exploited in consecutive layers in neural networks. Given two layers, $\mathbf{h}=f(\mathbf{W}^{(1)}\mathbf{x}+\mathbf{b}^{(1)})$ and $\mathbf{y}=f(\mathbf{W}^{(2)}\mathbf{h}+\mathbf{b}^{(2)})$, through scaling equivariance we have that:

[TABLE]

where $\mathbf{S}=diag(\mathbf{s})$ is a diagonal matrix with value $\mathbf{S}_{ii}$ denoting the scaling factor $\mathbf{s}_{i}$ for neuron $i$. This allows us to reparameterize our model with $\mathbf{\widehat{W}}^{(2)}=\mathbf{W}^{(2)}\mathbf{S}$, $\mathbf{\widehat{W}}^{(1)}=\mathbf{S^{-1}}\mathbf{W}^{(1)}$ and $\mathbf{\widehat{b}}^{(1)}=\mathbf{S^{-1}}\mathbf{b}^{(1)}$. In case of CNNs the scaling will be per channel and broadcast accordingly over the spatial dimensions. The rescaling procedure is illustrated in Figure 5.

4.1.2 Equalizing ranges over multiple layers

We can exploit the rescaling and reparameterization of the model to make the model more robust to quantization. Ideally the ranges of each channel $i$ are equal to the total range of the weight tensor, meaning we use the best possible representative power per channel. We define the precision of a channel as:

[TABLE]

where $\mathbf{\hat{r}}_{i}^{(1)}$ is the quantization range of channel $i$ in $\mathbf{\widehat{W}}^{(1)}$ and $\hat{R}^{(1)}$ is the total range of $\mathbf{\widehat{W}}^{(1)}$. We want to find $\mathbf{S}$ such that the total precision per channel is maximized:

[TABLE]

In the case of symmetric quantization we have $\mathbf{\hat{r}}_{i}^{(1)}=2\cdot\max_{j}|\mathbf{\widehat{W}_{ij}^{(1)}}|$ and $\hat{R}^{(1)}=2\cdot\max_{ij}|\mathbf{\widehat{W}_{ij}^{(1)}}|$. Solving eq. 9 (see appendix A) leads to the necessary condition:

[TABLE]

meaning the limiting channel defining the quantization range is given by $\operatorname*{arg\,max}_{i}\mathbf{r}_{i}^{(1)}\mathbf{r}_{i}^{(2)}$. We can satisfy this condition by setting $\mathbf{S}$ such that:

[TABLE]

which results in $\forall i:\mathbf{r}_{i}^{(1)}=\mathbf{r}_{i}^{(2)}$. Thus the channel’s ranges between both tensors are matched as closely as possible.

When equalizing multiple layers at the same time, we iterate this process for pairs of layers that are connected to each other without input or output splits in between, until convergence.

4.1.3 Absorbing high biases

In case $\mathbf{s}_{i}<1$ the equalization procedure increases bias $\mathbf{b}^{(1)}_{i}$. This could in turn increase the range of the activation quantization. In order to avoid big differences between per-channel ranges in the activations we introduce a procedure that absorbs high biases into the subsequent layer.

For a layer with ReLU function $r$, there is a non-negative vector $\mathbf{c}$ such that $r(\mathbf{W}\mathbf{x}+\mathbf{b}-\mathbf{c})=r(\mathbf{W}\mathbf{x}+\mathbf{b})-\mathbf{c}$. The trivial solution $\mathbf{c}=\mathbf{0}$ holds for all $\mathbf{x}$. However, depending on the distribution of $\mathbf{x}$ and the values of $\mathbf{W}$ and $\mathbf{b}$, there can be some values $\mathbf{c}_{i}>\mathbf{0}$ for which this equality holds for (almost) all $\mathbf{x}$. Following the previous two layer example, these $\mathbf{c}_{i}$ can be absorbed from layer $1$ into layer $2$ as:

[TABLE]

where $\mathbf{\hat{b}}^{(2)}=\mathbf{W}^{(2)}\mathbf{c}+\mathbf{b}^{(2)}$, $\mathbf{\hat{h}}=\mathbf{h}-\mathbf{c}$, and $\mathbf{\hat{b}}^{(1)}=\mathbf{b}^{(1)}-\mathbf{c}$.

To find $\mathbf{c}$ without violating our data-free assumption we assume that the pre-bias activations are distributed normally with the batch normalization shift and scale parameters $\bm{\beta}$ and $\bm{\gamma}$ as its mean and standard deviation. We set $\mathbf{c}=\max(\mathbf{0},\bm{\beta}-3\bm{\gamma})$. If $\mathbf{c}>0$, the equality introduced above will hold for the $99.865\%$ of values of $\mathbf{x}$ (those greater than $\mathbf{c}$) under the Gaussian assumption. As we will show in section 5.1.1, this approximation does not harm the full precision performance significantly but helps for activation quantization. Note that, in case data is available, the pre-bias distribution of $\mathbf{x}$ can be found empirically and used to set $\mathbf{c}$.

4.2 Quantization bias correction

As shown empirically in the motivation, quantization can introduce a biased error in the activations. In this section we show how to correct for the bias in the error on the layer’s output, and how we can use the network’s batch normalization parameters to compute this bias without using data.

For a fully connected layer with weight tensor $\mathbf{W}$, quantized weights $\mathbf{\widetilde{W}}$, and input activations $\mathbf{x}$, we have $\mathbf{\widetilde{y}}=\mathbf{\widetilde{W}}\mathbf{x}$ and therefore $\mathbf{\widetilde{y}}=\mathbf{y}+\bm{\epsilon}\mathbf{x}$, where we define the quantization error $\bm{\epsilon}=\mathbf{\widetilde{W}}-\mathbf{W}$, $\mathbf{y}$ as the layer pre-activations of the FP32 model, and $\mathbf{\widetilde{y}}$ that layer with quantization error added.

If the expectation of the error for output $i$, $\mathbb{E}[\bm{\epsilon}\textbf{x}]_{i}\neq 0$, then the mean of the output $i$ will change. This shift in distribution may lead to detrimental behavior in the following layers. We can correct for this change by seeing that:

[TABLE]

Thus, subtracting the expected error on the output $\mathbb{E}\left[\bm{\epsilon}\mathbf{x}\right]=\bm{\epsilon}\mathbb{E}\left[\mathbf{x}\right]$ from the biased output $\mathbf{\widetilde{y}}$ ensures that the mean for each output unit is preserved.

For implementation, the expected error can be subtracted from the layer’s bias parameter, since the expected error vector has the same shape as the layer’s output. This method easily extends to convolutional layers as described in Appendix B.

4.2.1 Computing the expected input

To compute the expected error of the output of a layer, the expected input to the layer $\mathbb{E}[\mathbf{x}]$ is required. If a model does not use batch normalization, or there are no data-usage restrictions, $\mathbb{E}[\bm{\epsilon}\mathbf{x}]$ can be computed by comparing the activations before and after quantization. Appendix D explains this procedure in more detail.

Clipped normal distribution

When the network includes batch normalization before a layer, we can use it to calculate $\mathbb{E}[\mathbf{x}]$ for that layer without using data. We assume the pre-activation outputs of a layer are normally distributed, that batch normalization is applied before the activation function, and that the activation function is some form of the class of clipped linear activation functions (e.g. ReLU, ReLU6), which clips its input range to the range $[a,b]$ where $a<b$, and $b$ can be $\infty$.

Due to the centralization and normalization applied by batch normalization, the mean and standard deviation of the pre-activations are known: these are the batch normalization scale and shift parameters (henceforth referred to as $\bm{\gamma}$ and $\bm{\beta}$ respectively).

To compute $\mathbb{E}[\mathbf{x}]$ from the previous layer’s batch normalization parameters, the mean and variance need to be adjusted to account for the activation function that follows the batch normalization layer. For this purpose we introduce the clipped normal distribution. A clipped-normally distributed random variable $X$ is a normally distributed random variable with mean $\mu$ and variance $\sigma^{2}$, whose values are clipped to the range $[a,b]$ The mean and variance of the clipped normal distribution can be computed in closed form from $\mu$, $\sigma$, $a$ and $b$. We present the mean of the clipped normal distribution for the ReLU activation function, i.e. $a=0$ and $b=\infty$ in this section, and refer the reader to Appendix C for the closed form solution for the general clipped normal distribution.

The expected value for channel $c$ in $\mathbf{x}$, $\mathbb{E}[\mathbf{x}_{c}]$, which is the output of a layer with batch normalization parameters $\bm{\beta}_{c}$ and $\bm{\gamma}_{c}$, followed by a ReLU activation function is:

[TABLE]

where $\mathbf{x}_{c}^{pre}$ is the pre-activation output for channel $c$, which is assumed to be normally distributed with mean $\bm{\beta}_{c}$ and variance $\bm{\gamma}_{c}^{2}$, $\Phi(\cdot)$ is the normal CDF, and the notation $\mathcal{N}(x)$ is used to denote the normal $\mathcal{N}(x|0,1)$ PDF.

5 Experiments

In this section we present two sets of experiments to validate the performance of data-free quantization (DFQ). We first show in section 5.1 the effect of the different aspects of DFQ and how they solve the problems observed earlier. Then we show in section 5.2 how DFQ generalizes to other models and tasks, and sets a new state-of-the-art for level 1 quantization.

To allow comparison to previously published results, we use both weights and activations are quantized using 8-bit asymmetric, per-tensor quantization in all experiments. Batch normalization is folded in the adjacent layer before quantization. Weight quantization ranges are the min and max of the weight tensor. Activation quantization ranges are set without data, by using the learned batch normalization shift and scale parameter vectors ${\bm{\beta}}$ and ${\bm{\gamma}}$ as follows: We compute the activation range for channel $i$ as ${\bm{\beta}}_{i}\pm n\cdot{\bm{\gamma}}_{i}$ (with $n=6$), with the minimum clipped to 0 in case of ReLU activation. We observed a wide range of $n$ can be used without significant performance difference. All experiments are done in Pytorch [26]. In appendix 8 we show additional experiments using short-term fine-tuning, symmetric quantization and per-channel quantization.

5.1 Ablation study

In this section we investigate the effect of our methods on a pre-trained MobileNetV2 [30] model111We use the Pytorch implementation of MobileNetV2 provided by https://github.com/tonylins/pytorch-mobilenet-v2.. We validate the performance of the model on the ImageNet [29] validation set. We first investigate the effects of different parts of our approach through a set of ablation studies.

5.1.1 Cross-layer equalization

In this section we investigate the effects of cross-layer equalization and high-bias folding. We compare these methods to two baselines: the original quantized model and the less hardware friendly per-channel quantization scheme.

The models considered in this section employ residual connections [12]. For these networks we apply cross-layer equalization only to the layers within each residual block. MobileNetV2 uses ReLU6 activation functions, which clips activation ranges to $[0,6]$. To avoid ReLU6 requiring a different cut off per channel after applying the equalization procedure, we replace ReLU6 with regular ReLU.

The results of the equalization experiments are shown in Table 1. Similar to [18], we observe that the model performance is close to random when quantizing the original model to INT8. Further we note that replacing ReLU6 by ReLU does not significantly degrade the model performance. Applying equalization brings us to within 2% of FP32 performance, close to the performance of per-channel quantization. We note that absorbing high biases results in a small drop in FP32 performance, but it boosts quantized performance by 1% due to more precise activation quantization. Combining both methods improves performance over per-channel quantization, indicating the more efficient per-tensor quantization could be used instead.

To illustrate the effect of cross-layer equalization, we show the weight distributions per output channel of the depthwise-separable layer in the model’s first inverted residual block after applying the equalization in Figure 6. We observe that most channels ranges are now similar and that the strong outliers from Figure 2 have been equalized. Note, there are still several channels which have all weight values close to zero. These channels convey little information and can be pruned from the network with hardly any loss in accuracy.

5.1.2 Bias correction

In this section we present results on bias correction for a quantized MobileNetV2 model. We furthermore present results of bias correction in combination with a naive weight-clipping baseline, and combined with the cross-layer equalization approach.

The weight-clipping baseline serves two functions: 1) as a naive baseline to the cross-layer equalization approach, and 2) to show that bias correction can be employed in any setting where biased noise is introduced. Weight clipping solves the problem of large differences in ranges between channels by clipping large ranges to smaller ranges, but it introduces a strongly biased error. Weight clipping is applied by first folding the batch normalization parameters into a layer’s weights, and then clipping all values to a certain range, in this case $[-15,15]$. We tried multiple symmetric ranges, all provided similar results. For residual connections we calculate $\mathbb{E}[\mathbf{x}]$ and $\text{Var}[\mathbf{x}]$ based on the sum and variance of all input expectations, taking the input to be zero mean and unit variance.

To illustrate the effect of bias correction, Figure 3 shows the per output channel biased error introduced by weight quantization. The per-channel biases are obtained as described in eq. 1. This figure shows that applying bias correction reduces the bias in the error on the output of a layer to very close to 0 for most output channels.

Results for the experiments described above for MobileNet V2 on the ImageNet validation set are shown in Table 2. Applying bias correction improves quantized model performance, indicating that a part of the problem of quantizing this model lies in the biased error that is introduced. However, bias correction on its own does not achieve near-floating point performance. The reason for this is most likely that the problem described in 3.1 is more severe for this model. The experiments on weight-clipping show that bias correction can mitigate performance degradation due to biased error in non-quantized models as well as quantized models. Clipping without correction in the FP32 model introduces a 4.66% loss in accuracy; bias correction reduces that loss to a mere 0.57%. Furthermore, it shows that weight clipping combined with bias correction is a fairly strong baseline for quantizing MobileNet V2. Lastly, we show that bias correction improves results when combined with the cross-layer equalization and bias folding procedures. The combination of all methods is our data-free quantization (DFQ) method. The full DFQ approach achieves near-floating point performance with a reduction of 0.53% top 1 accuracy relative to the FP32 baseline.

5.2 Comparison to other methods and models

In this section we show how DFQ generalizes to other popular computer vision tasks, namely semantic segmentation and object detection, and other model architectures such as MobileNetV1 [14] and Resnet18 [12]. Afterwards we compare DFQ to methods in the literature, including more complex level 3 and 4 approaches. This set of models was chosen as they are efficient and likely to be used in mobile applications where 8-bit quantization is frequently used for power efficiency.

5.2.1 Other tasks

Semantic segmentation

To demonstrate the generalization of our method to semantic segmentation we apply DFQ for DeeplabV3+ with a MobileNetV2 backend [3, 30], performance is evaluated on the Pascal VOC segmentation challenge [6]. For our experiments we use the publicly available Pytorch implementation222https://github.com/jfzhang95/pytorch-deeplab-xception.

We show the results of this experiment in Table 3. As observed earlier for classification we notice a significant drop in performance when quantizing the original model which makes it almost unusable in practice. Applying DFQ recovers almost all performance degradation and achieves less than 1% drop in mIOU compared to the full precision model. DFQ also outperforms the less hardware friendly per-channel quantization. To the best of our knowledge we are the first to publish quantization results on DeeplabV3+ as well as for semantic segmentation.

Object detection

To demonstrate the applicability of our method to object detection we apply DFQ for MobileNetV2 SSDLite [30, 20], evaluated on the Pascal VOC object detection challenge [6]. In our experiments we use the publicly available Pytorch implementation of SSD333https://github.com/qfgaohao/pytorch-ssd.

The results are listed in Table 4. Similar to semantic segmentation we observe a significant drop in performance when quantizing the SSDLite model. Applying DFQ recovers almost all performance drop and achieves less than 1% drop in mAP compared to the full precision model, again outperforming per-channel quantization.

5.2.2 Comparison to other approaches

In this section we compare DFQ to other approaches in literature. We compare our results to two other level 1 approaches, direct per-layer quantization as well as per-channel quantization [18]. In addition we also compare to multiple higher level approaches, namely quantization aware training [16] as well as stochastic rounding and dynamic ranges [9, 10], which are both level 3 approaches. We also compare to two level 4 approaches based on relaxed quantization [21], which involve training a model from scratch and to quantization friendly separable convolutions [31] that require a rework of the original MobileNet architecture. The results are summarized in Table 5.

For both MobileNetV1 and MobileNetV2 per-layer quantization results in an unusable model whereas DFQ stays close to full precision performance. DFQ also outperforms per-channel quantization as well as most level 3 and 4 approaches which require significant fine-tuning, training or even architecture changes.

On Resnet18 we maintain full precision performance for 8-bit fixed point quantization using DFQ. Some higher level approaches [16, 21] report slightly higher results than our baseline model, likely due to a better training procedure than used in the standard Pytorch Resnet18 model. Since 8-bit quantization is lossless we also compare 6-bit results. DFQ clearly outperforms traditional per-layer quantization but stays slightly below per-channel quantization and higher level approaches such as QT and RQ [16, 21].

Overall DFQ sets a new state-of-the-art for 8-bit fixed point quantization on several models and computer vision tasks. It is especially strong for mobile friendly architectures such as MobileNetV1 and MobileNetV2 which were previously hard to quantize. Even though DFQ is an easy to use level 1 approach, we generally show competitive performance when comparing to more complex level 2-4 approaches.

6 Conclusion

In this work, we introduced DFQ, a data-free quantization method that significantly helps quantized model performance without the need for data, fine-tuning or hyper-parameter optimization. The method can be applied to many common computer vision architectures with a straight-forward API call. This is crucial for many practical applications where engineers want to deploy deep learning models trained in FP32 to INT8 hardware without much effort. Results are presented for common computer vision tasks like image classification, semantic segmentation and object detection. We show that our method compares favorably to per-channel quantization [18], meaning that instead the more efficient per-tensor quantization can be employed in practice. DFQ achieves near original model accuracy for almost every model we tested, and even competes with more complicated training based methods.

Further we introduced a set of quantization levels to facilitate the discussion on the applicability of quantization methods. There is a difference in how easy a method is to use for generating a quantized model, which is a significant part of the impact potential of a quantization method in real world applications. We hope that the quantization levels and methods introduced in this paper will contribute to both future research and practical deployment of quantized deep learning models.

Acknowledgments

We would like to thank Christos Louizos, Harris Teague, Jakub Tomczak, Mihir Jain and Pim de Haan for their helpful discussions and valuable feedback.

Appendix A Optimal range equalization of two layers

Consider two fully-connected layers with weight matrices $\mathbf{W}^{(1)}$ and $\mathbf{W}^{(2)}$, that we scale as in 4.1. We investigate the problem of optimizing the quantization ranges by rescaling the weight matrices by $\mathbf{S}=diag(\mathbf{s})$, where $\mathbf{s}>0$, such that $\mathbf{\widehat{W}}^{(1)}=\mathbf{S}^{-1}\mathbf{W}^{(1)}$ and $\mathbf{\widehat{W}}^{(2)}=\mathbf{W}^{(2)}\mathbf{S}$ the weight matrices after rescaling. We investigate the case of symmetric quantization, which also gives good results in practice for asymmetric quantization. We denote

[TABLE]

where $\mathbf{\hat{r}}^{(k)}$ are the per-channel weight ranges that are scaled by $\mathbf{S}$, $\hat{R}^{(k)}$ the range for the scaled weight matrix $\mathbf{\widehat{W}}^{(k)}$ and ${\mathbf{r}}^{(k)}$ are the original unscaled ranges.

Using this in our optimization goal of eq. 9 leads to

[TABLE]

We observe that the specific scaling $\mathbf{s}_{i}$ of each channel cancels out as long as they do not increase $R$, the range of the full weight matrix. We can reformulate the above to

[TABLE]

which is minimal iff

[TABLE]

By contradiction, if $j\neq k$ there is a small positive $\epsilon$ such that $\mathbf{s}^{\prime}_{k}=\mathbf{s}_{k}-\epsilon$ which will decrease $\max_{k}\mathbf{s}_{k}\mathbf{r}_{k}^{(2)}$ by $\epsilon\mathbf{r}_{k}^{(2)}$ without affecting $\max_{j}\frac{1}{\mathbf{s}_{j}}\mathbf{r}_{j}^{(1)}$. Therefore such a solution would not be optimal for eq. 26.

The condition from eq. 27 implies there is a limiting channel $i=\operatorname*{arg\,max}_{i}\mathbf{r}_{i}^{(1)}\mathbf{r}_{i}^{(2)}$ which defines the quantization range of both weight matrices $\mathbf{\widehat{W}}^{(1)}$ and $\mathbf{\widehat{W}}^{(2)}$. However, our optimization goal is not affected by the choice of the other $\mathbf{s}_{j}$ given the resulting $\mathbf{\hat{r}}_{j}^{(1)}$ and $\mathbf{\hat{r}}_{j}^{(2)}$ are smaller than or equal to $\mathbf{\hat{r}}_{i}^{(1)}$ and $\mathbf{\hat{r}}_{i}^{(2)}$, respectively. To break the ties of solutions we decide to set $\forall i:\mathbf{r}_{i}^{(1)}=\mathbf{r}_{i}^{(2)}$. Thus the channel’s ranges between both tensors are matched as closely as possible and the introduced quantization error is spread equally among both weight tensors. This results in our final rescaling factor

[TABLE]

which satisfies our necessary condition from eq. 27 and ensures that $\forall i:\mathbf{r}_{i}^{(1)}=\mathbf{r}_{i}^{(2)}$.

Appendix B Bias correction for convolutional layers

Similarly to fully connected layers we can compute $\bm{\epsilon}$ from $\mathbf{W}$ and $\mathbf{\widetilde{W}}$, it becomes a constant and we have that $\mathbb{E}\left[\bm{\epsilon}*\mathbf{x}\right]=\bm{\epsilon}*\mathbb{E}[\mathbf{x}]$. Expanding this yields:

[TABLE]

where we assume that the expected value of each input channel is the same for all spatial dimensions in the input channel. Since the value of $\left[\bm{\epsilon}*\mathbb{E}[\mathbf{x}]\right]_{c_{o}ij}$ does not depend on the spatial dimensions $i$ and $j$, the expected error is the same for the full output channel and can be folded into the layer’s bias parameter.

Appendix C Clipped normal distribution

Given a normally distributed random variable $X$ with mean $\mu$ and variance $\sigma^{2}$, and a clipped-linear function $f(\cdot)$ that clips its argument to the range $[a,b]$, s.t. $a<b$, the mean and variance of $f(X)$ can be determined using the standard rules of computing the mean and variance of a function:

[TABLE]

where we define $p(x)=\mathcal{N}(x\mid\mu,\sigma)$, ${\mu^{c}_{ab}}=\mathbb{E}[f(X)]$ and ${\sigma^{c}_{ab}}^{2}=Var[f(X)]$.

C.1 Mean of Clipped Normal Distribution

Using the fact that $f(x)$ is constant if $x\not\in[a,b]$ we have that:

[TABLE]

The first and last term can be computed as $a\Phi(\alpha)$ and $b(1-\Phi(\beta))$ respectively, where we define $\alpha=\frac{a-\mu}{\sigma}$, $\beta=\frac{b-\mu}{\sigma}$, and $\Phi(x)=CDF(x\mid 0,1)$, the normal CDF with zero mean and unit variance.

The integral over the linear part of $f(\cdot)$ can be computed as:

[TABLE]

where we define $\phi(\cdot)=\mathcal{N}(\cdot\mid 0,1)$, i.e. the standard normal pdf and $C=\frac{1}{\sigma\sqrt{2\pi}}$ is the normalization constant for a normal distribution with variance $\sigma^{2}$, thus

[TABLE]

C.2 Variance of Clipped Normal Distribution

We again exploit the fact that $f(x)$ is constant if $x\not\in[a,b]$:

[TABLE]

The first and last term can be solved as $(a-{\mu^{c}_{ab}})^{2}\Phi(\alpha)$ and $(b-{\mu^{c}_{ab}})^{2}(1-\Phi(\beta))$ respectively.

The second term can be decomposed as follows:

[TABLE]

where we use the result from the previous subsection and define $Z=\Phi(\beta)-\Phi(\alpha)$, and where ${\mu^{t}_{ab}}=\frac{1}{Z}\int_{a}^{b}x\mathcal{N}(x\mid\mu,\sigma^{2})=\mu+\sigma(\phi(\alpha)-\phi(\beta))/Z$ is the mean of the truncated normal distribution.

Evaluating the first term yields:

[TABLE]

This results in:

[TABLE]

Appendix D Empirical quantization bias correction

If a network does not use batch normalization, or does not use batch normalization in all layers, a representative dataset can be used to compute the difference between pre-activation means before and after quantization. We then subtract this difference from the quantized model’s pre-activations. This procedure can be run with unlabeled data. The procedure should be run after BatchNorm folding and cross-layer range equalization. Clipping should be applied in the quantized network, but not in the floating point network. Since the activation function and the quantization operation are fused, this procedure is run on a network with quantized weights only. However, after this procedure is applied activations can be quantized as well. We bias correct a layer only after all the layers feeding into it have been bias-corrected. The procedure is as follows:

Run $N$ examples through the FP32 model and collect for each layer the per-channel pre-activation means $\mathbb{E}[\mathbf{y}]$. 2. 2.

For each layer $L$ in the quantized model:

• •

Collect the per-channel pre-activation means $\mathbb{E}[\mathbf{\widetilde{y}}]$ of layer $L$ for the same $N$ examples as in step 1.

• •

Compute the per-channel biased quantization error $\mathbb{E}[\bm{\epsilon}]=\mathbb{E}[\mathbf{\widetilde{y}}]-\mathbb{E}[\mathbf{y}]$.

• •

Subtract $\mathbb{E}[\bm{\epsilon}]$ from the layer’s bias parameter.

In Table 6 we compare this empirical bias correction procedure with the analytic bias correction introduced in section 4.2. We observe that both approaches leads to similar results.

Appendix E Additional experiments

Combination with fine-tuning

The focus of our method is data-free quantization (level 1). However, our method can also be used as a pre-processing before quantization aware fine-tuning. To demonstrate this we used DFQ together with short-term quantization aware fine-tuning [18]. After just 1 epoch of quantization aware fine-tuning MobileNet V2, accuracy increases from 71.19% to 71.42%, almost recovering the FP32 performance (71.72%).

Symmetric vs asymmetric quantization

In our experimental section all our experiments were performed with asymmetric quantization, since this is commonly used in literature. Here we also compare to symmetric quantization. Symmetric quantization does not use an offset, which eliminates several cross terms in the calculations done on hardware compared to asymmetric quantization. This makes symmetric quantization more efficient on some hardware at the expense of losing some expressive power.

In Table 7 we compare symmetric and asymmetric quantization in combination with DFQ. For all three models the advantage of asymmetric quantization is almost negligible. We noticed that cross-layer equalization is effective at removing outliers, resulting in weight distributions are often close to symmetric.

DFQ combined with per-channel quantization

In our experiments we focused on per-tensor quantization since the more recent per-channel quantization is not efficiently supported on all hardware. For hardware that does support it, we analyze the effect of DFQ in combination with per-channel quantization.

In Table 8 we show the results of the different components of DFQ in combination with per-channel quantization. We notice that each individual component, cross-layer equalization, bias absorption and bias correction, incremental improve over per-channel quantization and reduce the total quantization error from 1.07% to only 0.39%.