ICE-Pruning: An Iterative Cost-Efficient Pruning Pipeline for Deep Neural Networks

TL;DR

ICE-Pruning introduces an efficient iterative pruning pipeline for deep neural networks that significantly reduces fine-tuning costs while maintaining accuracy, through automatic fine-tuning decisions, freezing strategies, and a custom learning rate scheduler.

Contribution

It proposes a novel ICE-Pruning pipeline with automatic fine-tuning triggers, a freezing strategy, and a pruning-aware learning rate scheduler to accelerate DNN pruning.

Findings

Accelerates pruning by up to 9.61x

Maintains similar accuracy to existing methods

Reduces overall fine-tuning time

Abstract

Pruning is a widely used method for compressing Deep Neural Networks (DNNs), where less relevant parameters are removed from a DNN model to reduce its size. However, removing parameters reduces model accuracy, so pruning is typically combined with fine-tuning, and sometimes other operations such as rewinding weights, to recover accuracy. A common approach is to repeatedly prune and then fine-tune, with increasing amounts of model parameters being removed in each step. While straightforward to implement, pruning pipelines that follow this approach are computationally expensive due to the need for repeated fine-tuning. In this paper we propose ICE-Pruning, an iterative pruning pipeline for DNNs that significantly decreases the time required for pruning by reducing the overall cost of fine-tuning, while maintaining a similar accuracy to existing pruning pipelines. ICE-Pruning is based on three main components: i) an automatic mechanism to determine after which pruning steps fine-tuning should be performed; ii) a freezing strategy for faster fine-tuning in each pruning step; and iii) a custom pruning-aware learning rate scheduler to further improve the accuracy of each pruning step and reduce the overall time consumption. We also propose an efficient auto-tuning stage for the hyperparameters (e.g., freezing percentage) introduced by the three components. We evaluate ICE-Pruning on several DNN models and datasets, showing that it can accelerate pruning by up to 9.61x. Code is available at https://github.com/gicLAB/ICE-Pruning