Neural Network Activation Quantization with Bitwise Information Bottlenecks

TL;DR

This paper introduces a novel bitwise information bottleneck method for neural activation quantization, optimizing rate-distortion trade-offs to achieve high accuracy with low-precision activations, significantly improving efficiency.

Contribution

It proposes a layer-wise quantization approach based on rate-distortion theory, enhancing neural network efficiency without sacrificing accuracy.

Findings

Achieves state-of-the-art accuracy with low-precision activations.

Improves memory and computational efficiency over six times.

Demonstrates effectiveness on ImageNet and other datasets.

Abstract

Recent researches on information bottleneck shed new light on the continuous attempts to open the black box of neural signal encoding. Inspired by the problem of lossy signal compression for wireless communication, this paper presents a Bitwise Information Bottleneck approach for quantizing and encoding neural network activations. Based on the rate-distortion theory, the Bitwise Information Bottleneck attempts to determine the most significant bits in activation representation by assigning and approximating the sparse coefficient associated with each bit. Given the constraint of a limited average code rate, the information bottleneck minimizes the rate-distortion for optimal activation quantization in a flexible layer-by-layer manner. Experiments over ImageNet and other datasets show that, by minimizing the quantization rate-distortion of each layer, the neural network with information bottlenecks achieves the state-of-the-art accuracy with low-precision activation. Meanwhile, by reducing the code rate, the proposed method can improve the memory and computational efficiency by over six times compared with the deep neural network with standard single-precision representation. Codes will be available on GitHub when the paper is accepted \url{https://github.com/BitBottleneck/PublicCode}.