Towards the Limit of Network Quantization

TL;DR

This paper develops a theoretically grounded approach to neural network quantization that minimizes performance loss under compression constraints, using Hessian-weighted clustering and entropy coding, achieving high compression ratios.

Contribution

It introduces a Hessian-weighted quantization scheme and links network quantization to entropy-constrained scalar quantization, providing new methods for efficient neural network compression.

Findings

Achieved compression ratios of 51.25, 22.17, and 40.65 for LeNet, ResNet, and AlexNet.

Proposed Hessian-weighted k-means clustering for optimal parameter quantization.

Linked network quantization to entropy-constrained scalar quantization in information theory.

Abstract

Network quantization is one of network compression techniques to reduce the redundancy of deep neural networks. It reduces the number of distinct network parameter values by quantization in order to save the storage for them. In this paper, we design network quantization schemes that minimize the performance loss due to quantization given a compression ratio constraint. We analyze the quantitative relation of quantization errors to the neural network loss function and identify that the Hessian-weighted distortion measure is locally the right objective function for the optimization of network quantization. As a result, Hessian-weighted k-means clustering is proposed for clustering network parameters to quantize. When optimal variable-length binary codes, e.g., Huffman codes, are employed for further compression, we derive that the network quantization problem can be related to the entropy-constrained scalar quantization (ECSQ) problem in information theory and consequently propose two solutions of ECSQ for network quantization, i.e., uniform quantization and an iterative solution similar to Lloyd's algorithm. Finally, using the simple uniform quantization followed by Huffman coding, we show from our experiments that the compression ratios of 51.25, 22.17 and 40.65 are achievable for LeNet, 32-layer ResNet and AlexNet, respectively.