Pruning CNN's with linear filter ensembles

TL;DR

This paper introduces a novel CNN pruning method based on filter importance norms derived from empirical loss changes, enabling significant parameter and FLOP reduction with minimal accuracy loss.

Contribution

The paper proposes a new filter importance measure based on empirical loss change and a sampling approach to create filter ensembles for effective pruning.

Findings

Removed 60% of parameters from ResNet on CIFAR-10

Achieved 64% FLOP reduction with less than 0.6% accuracy drop

Outperformed traditional norm-based pruning methods

Abstract

Despite the promising results of convolutional neural networks (CNNs), their application on devices with limited resources is still a big challenge; this is mainly due to the huge memory and computation requirements of the CNN. To counter the limitation imposed by the network size, we use pruning to reduce the network size and -- implicitly -- the number of floating point operations (FLOPs). Contrary to the filter norm method -- used in ``conventional`` network pruning -- based on the assumption that a smaller norm implies ``less importance'' to its associated component, we develop a novel filter importance norm that is based on the change in the empirical loss caused by the presence or removal of a component from the network architecture. Since there are too many individual possibilities for filter configuration, we repeatedly sample from these architectural components and measure the system performance in the respective state of components being active or disabled. The result is a collection of filter ensembles -- filter masks -- and associated performance values. We rank the filters based on a linear and additive model and remove the least important ones such that the drop in network accuracy is minimal. We evaluate our method on a fully connected network, as well as on the ResNet architecture trained on the CIFAR-10 dataset. Using our pruning method, we managed to remove $60\%$ of the parameters and $64\%$ of the FLOPs from the ResNet with an accuracy drop of less than $0.6\%$.