And the Bit Goes Down: Revisiting the Quantization of Neural Networks

TL;DR

This paper presents a vector quantization technique that reduces the memory footprint of neural networks by focusing on output reconstruction quality, enabling efficient inference with significant compression while maintaining high accuracy.

Contribution

We propose a novel output-focused vector quantization method for neural network compression that requires only unlabelled data and supports efficient CPU inference.

Findings

ResNet-50 compressed to 5MB with 76.1% top-1 accuracy

Mask R-CNN compressed by 26x while preserving performance

Method achieves high compression with minimal accuracy loss

Abstract

In this paper, we address the problem of reducing the memory footprint of convolutional network architectures. We introduce a vector quantization method that aims at preserving the quality of the reconstruction of the network outputs rather than its weights. The principle of our approach is that it minimizes the loss reconstruction error for in-domain inputs. Our method only requires a set of unlabelled data at quantization time and allows for efficient inference on CPU by using byte-aligned codebooks to store the compressed weights. We validate our approach by quantizing a high performing ResNet-50 model to a memory size of 5MB (20x compression factor) while preserving a top-1 accuracy of 76.1% on ImageNet object classification and by compressing a Mask R-CNN with a 26x factor.