Automatic Neural Network Pruning that Efficiently Preserves the Model Accuracy

TL;DR

This paper introduces an automatic neural network pruning method that efficiently reduces FLOPs while preserving accuracy, requiring only one epoch of training and outperforming existing techniques across various architectures and datasets.

Contribution

The proposed method uses a trainable bottleneck to automatically select neurons to prune, significantly reducing training time and maintaining high accuracy.

Findings

52% FLOPs reduction on ResNet-50 with 47.51% accuracy

State-of-the-art accuracy of 76.63% on ILSVRC2012 after finetuning

Requires only one epoch and a small dataset subset for pruning

Abstract

Neural networks performance has been significantly improved in the last few years, at the cost of an increasing number of floating point operations per second (FLOPs). However, more FLOPs can be an issue when computational resources are limited. As an attempt to solve this problem, pruning filters is a common solution, but most existing pruning methods do not preserve the model accuracy efficiently and therefore require a large number of finetuning epochs. In this paper, we propose an automatic pruning method that learns which neurons to preserve in order to maintain the model accuracy while reducing the FLOPs to a predefined target. To accomplish this task, we introduce a trainable bottleneck that only requires one single epoch with 25.6% (CIFAR-10) or 7.49% (ILSVRC2012) of the dataset to learn which filters to prune. Experiments on various architectures and datasets show that the proposed method can not only preserve the accuracy after pruning but also outperform existing methods after finetuning. We achieve a 52.00% FLOPs reduction on ResNet-50, with a Top-1 accuracy of 47.51% after pruning and a state-of-the-art (SOTA) accuracy of 76.63% after finetuning on ILSVRC2012. Code available at https://github.com/nota-github/autobot_AAAI23.