Reproducibility Study: Comparing Rewinding and Fine-tuning in Neural Network Pruning

TL;DR

This study reproduces and evaluates three approaches for retraining pruned neural networks, highlighting the limitations of the newly proposed learning rate rewinding method on larger architectures.

Contribution

It verifies the original results and extends the analysis to larger networks, revealing limitations of the learning rate rewinding approach.

Findings

Exact reproduction of original results across scenarios

Learning rate rewinding can degrade accuracy on large architectures

The core conclusions of the original paper are confirmed

Abstract

Scope of reproducibility: We are reproducing Comparing Rewinding and Fine-tuning in Neural Networks from arXiv:2003.02389. In this work the authors compare three different approaches to retraining neural networks after pruning: 1) fine-tuning, 2) rewinding weights as in arXiv:1803.03635 and 3) a new, original method involving learning rate rewinding, building upon Lottery Ticket Hypothesis. We reproduce the results of all three approaches, but we focus on verifying their approach, learning rate rewinding, since it is newly proposed and is described as a universal alternative to other methods. We used CIFAR10 for most reproductions along with additional experiments on the larger CIFAR100, which extends the results originally provided by the authors. We have also extended the list of tested network architectures to include Wide ResNets. The new experiments led us to discover the limitations of learning rate rewinding which can worsen pruning results on large architectures. Results: We were able to reproduce the exact results reported by the authors in all originally reported scenarios. However, extended results on larger Wide Residual Networks have demonstrated the limitations of the newly proposed learning rate rewinding -- we observed a previously unreported accuracy degradation for low sparsity ranges. Nevertheless, the general conclusion of the paper still holds and was indeed reproduced.