Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression

TL;DR

This paper proposes a novel, architecture-agnostic method for neural network compression using cross-layer shared representations and additive quantization, achieving high compression ratios without accuracy loss.

Contribution

It introduces a simple, effective compression scheme that leverages shared parameter representations across layers, outperforming traditional pruning methods.

Findings

Achieves up to 15.3x compression with minimal accuracy loss.

Shared representations often occur across network layers.

Outperforms iterative unstructured pruning in experiments.

Abstract

This paper investigates deep neural network (DNN) compression from the perspective of compactly representing and storing trained parameters. We explore the previously overlooked opportunity of cross-layer architecture-agnostic representation sharing for DNN parameters. To do this, we decouple feedforward parameters from DNN architectures and leverage additive quantization, an extreme lossy compression method invented for image descriptors, to compactly represent the parameters. The representations are then finetuned on task objectives to improve task accuracy. We conduct extensive experiments on MobileNet-v2, VGG-11, ResNet-50, Feature Pyramid Networks, and pruned DNNs trained for classification, detection, and segmentation tasks. The conceptually simple scheme consistently outperforms iterative unstructured pruning. Applied to ResNet-50 with 76.1% top-1 accuracy on the ILSVRC12 classification challenge, it achieves a $7.2\times$ compression ratio with no accuracy loss and a $15.3\times$ compression ratio at 74.79% accuracy. Further analyses suggest that representation sharing can frequently happen across network layers and that learning shared representations for an entire DNN can achieve better accuracy at the same compression ratio than compressing the model as multiple separate parts. We release PyTorch code to facilitate DNN deployment on resource-constrained devices and spur future research on efficient representations and storage of DNN parameters.