Quantization-Aware Regularizers for Deep Neural Networks Compression

TL;DR

This paper introduces a novel regularization approach during training that encourages neural network weights to naturally cluster, making quantization more accurate and integrated into the learning process, thereby improving compression without significant accuracy loss.

Contribution

It presents a new regularization method that embeds quantization awareness into training, allowing weights to form clusters and quantization parameters to be learned during backpropagation.

Findings

Effective on CIFAR-10 with AlexNet and VGG16

Reduces accuracy loss in quantization

Integrates quantization into training process

Abstract

Deep Neural Networks reached state-of-the-art performance across numerous domains, but this progress has come at the cost of increasingly large and over-parameterized models, posing serious challenges for deployment on resource-constrained devices. As a result, model compression has become essential, and -- among compression techniques -- weight quantization is largely used and particularly effective, yet it typically introduces a non-negligible accuracy drop. However, it is usually applied to already trained models, without influencing how the parameter space is explored during the learning phase. In contrast, we introduce per-layer regularization terms that drive weights to naturally form clusters during training, integrating quantization awareness directly into the optimization process. This reduces the accuracy loss typically associated with quantization methods while preserving their compression potential. Furthermore, in our framework quantization representatives become network parameters, marking, to the best of our knowledge, the first approach to embed quantization parameters directly into the backpropagation procedure. Experiments on CIFAR-10 with AlexNet and VGG16 models confirm the effectiveness of the proposed strategy.