SplitQuant: Layer Splitting for Low-Bit Neural Network Quantization

TL;DR

SplitQuant is a novel layer splitting method that improves low-bit neural network quantization by better handling outliers, leading to higher accuracy in quantized models like BERT-Tiny.

Contribution

The paper introduces SplitQuant, a new layer splitting technique that preserves outliers and enhances quantization resolution for low-bit neural networks.

Findings

Improved INT2 quantization accuracy by up to 3.3 percentage points.

Achieved quantized model accuracy comparable to FP32 models.

Effective on BERT-Tiny models with minimal accuracy loss.

Abstract

Quantization for deep neural networks (DNNs) is the process of mapping the parameter values of DNNs from original data types to other data types of lower precision to reduce model sizes and make inference faster. Quantization often maps different original values to a single quantized value because the range of the original values is larger than the range of the quantized values. This leads to the degradation of the accuracy of the quantized DNNs. Outliers are a main cause of the degradation of quantization resolution because they enlarge the range of original values. To solve the problem, the percentile method is often used to clip outliers. However, clipping the outliers has another problem of removing the important and strong signals in the DNNs. This paper proposes SplitQuant to keep the outliers and improve the quantization resolution at the same time. SplitQuant narrows down the range of the original values and mitigates the effect of outliers by splitting each quantizable layer into three mathematically equivalent layers and applies different scaling factors. Especially, weights and biases are clustered into lower, middle and upper clusters for optimized split. By preprocessing DNNs with SplitQuant, quantization algorithms can achieve better results. SplitQuant was applied on two BERT-Tiny models and improved the accuracy of INT2 quantization by 3.3%p and 2.1%p, achieving accuracies comparable to those of the original FP32 models.