Cross-filter compression for CNN inference acceleration

TL;DR

This paper introduces a novel cross-filter compression technique for CNNs that significantly reduces memory and computation requirements by sharing scaling factors among adjacent filters, enabling faster inference with minimal accuracy loss.

Contribution

The paper proposes a new cross-filter compression method that surpasses traditional filter-wise quantization limits, achieving substantial memory and speed improvements for CNN inference.

Findings

~32x memory savings achieved

122x speedup in convolution operations

Tolerable accuracy loss on CIFAR-10 and ImageNet

Abstract

Convolution neural network demonstrates great capability for multiple tasks, such as image classification and many others. However, much resource is required to train a network. Hence much effort has been made to accelerate neural network by reducing precision of weights, activation, and gradient. However, these filter-wise quantification methods exist a natural upper limit, caused by the size of the kernel. Meanwhile, with the popularity of small kernel, the natural limit further decrease. To address this issue, we propose a new cross-filter compression method that can provide $\sim32\times$ memory savings and $122\times$ speed up in convolution operations. In our method, all convolution filters are quantized to given bits and spatially adjacent filters share the same scaling factor. Our compression method, based on Binary-Weight and XNOR-Net separately, is evaluated on CIFAR-10 and ImageNet dataset with widely used network structures, such as ResNet and VGG, and witness tolerable accuracy loss compared to state-of-the-art quantification methods.