How I Learned to Stop Worrying and Love Retraining

TL;DR

This paper demonstrates that a simple linear learning rate schedule can significantly shorten retraining in neural network pruning, outperforming complex methods and challenging the need to avoid retraining altogether.

Contribution

It introduces an adaptive linear learning rate schedule for retraining, showing that efficient pruning can be achieved within fixed training budgets and questioning traditional pruning practices.

Findings

Shortened retraining phase with linear schedule outperforms complex methods.

Adaptive schedule selection improves pruning efficiency.

Incorporating sparsification into standard training can reduce the need for retraining.

Abstract

Many Neural Network Pruning approaches consist of several iterative training and pruning steps, seemingly losing a significant amount of their performance after pruning and then recovering it in the subsequent retraining phase. Recent works of Renda et al. (2020) and Le & Hua (2021) demonstrate the significance of the learning rate schedule during the retraining phase and propose specific heuristics for choosing such a schedule for IMP (Han et al., 2015). We place these findings in the context of the results of Li et al. (2020) regarding the training of models within a fixed training budget and demonstrate that, consequently, the retraining phase can be massively shortened using a simple linear learning rate schedule. Improving on existing retraining approaches, we additionally propose a method to adaptively select the initial value of the linear schedule. Going a step further, we propose similarly imposing a budget on the initial dense training phase and show that the resulting simple and efficient method is capable of outperforming significantly more complex or heavily parameterized state-of-the-art approaches that attempt to sparsify the network during training. These findings not only advance our understanding of the retraining phase, but more broadly question the belief that one should aim to avoid the need for retraining and reduce the negative effects of 'hard' pruning by incorporating the sparsification process into the standard training.