Filter Distillation for Network Compression

TL;DR

This paper introduces Principal Filter Analysis (PFA), a practical method for neural network compression that leverages filter response correlations to reduce model size while maintaining accuracy across various architectures and datasets.

Contribution

The paper presents PFA, a novel, easy-to-use compression technique that adapts networks to specific domains and properties by analyzing filter response correlations.

Findings

Achieves up to 8x compression on VGG-16 with minimal accuracy loss.

Outperforms or matches state-of-the-art compression methods.

Demonstrates flexibility and practicality across multiple datasets and architectures.

Abstract

In this paper we introduce Principal Filter Analysis (PFA), an easy to use and effective method for neural network compression. PFA exploits the correlation between filter responses within network layers to recommend a smaller network that maintain as much as possible the accuracy of the full model. We propose two algorithms: the first allows users to target compression to specific network property, such as number of trainable variable (footprint), and produces a compressed model that satisfies the requested property while preserving the maximum amount of spectral energy in the responses of each layer, while the second is a parameter-free heuristic that selects the compression used at each layer by trying to mimic an ideal set of uncorrelated responses. Since PFA compresses networks based on the correlation of their responses we show in our experiments that it gains the additional flexibility of adapting each architecture to a specific domain while compressing. PFA is evaluated against several architectures and datasets, and shows considerable compression rates without compromising accuracy, e.g., for VGG-16 on CIFAR-10, CIFAR-100 and ImageNet, PFA achieves a compression rate of 8x, 3x, and 1.4x with an accuracy gain of 0.4%, 1.4% points, and 2.4% respectively. Our tests show that PFA is competitive with state-of-the-art approaches while removing adoption barriers thanks to its practical implementation, intuitive philosophy and ease of use.