One-Cycle Pruning: Pruning ConvNets Under a Tight Training Budget

TL;DR

This paper introduces One-Cycle Pruning, a novel method that integrates pruning and training into a single cycle, significantly reducing training time while improving the performance of sparsified ConvNets.

Contribution

The paper proposes a new pruning schedule that combines pruning and training, eliminating the need for a separate convergence training phase, and demonstrates its effectiveness across multiple architectures and datasets.

Findings

Outperforms existing pruning schedules like One-Shot and Iterative Pruning.

Achieves high sparsity levels (80-95%) with better accuracy.

Reduces training budget for pruning significantly.

Abstract

Introducing sparsity in a neural network has been an efficient way to reduce its complexity while keeping its performance almost intact. Most of the time, sparsity is introduced using a three-stage pipeline: 1) train the model to convergence, 2) prune the model according to some criterion, 3) fine-tune the pruned model to recover performance. The last two steps are often performed iteratively, leading to reasonable results but also to a time-consuming and complex process. In our work, we propose to get rid of the first step of the pipeline and to combine the two other steps in a single pruning-training cycle, allowing the model to jointly learn for the optimal weights while being pruned. We do this by introducing a novel pruning schedule, named One-Cycle Pruning, which starts pruning from the beginning of the training, and until its very end. Adopting such a schedule not only leads to better performing pruned models but also drastically reduces the training budget required to prune a model. Experiments are conducted on a variety of architectures (VGG-16 and ResNet-18) and datasets (CIFAR-10, CIFAR-100 and Caltech-101), and for relatively high sparsity values (80%, 90%, 95% of weights removed). Our results show that One-Cycle Pruning consistently outperforms commonly used pruning schedules such as One-Shot Pruning, Iterative Pruning and Automated Gradual Pruning, on a fixed training budget.