Cyclical Pruning for Sparse Neural Networks

TL;DR

This paper introduces cyclical pruning, a novel method for sparse neural network training that periodically prunes and recovers weights, outperforming existing methods especially at high sparsity levels.

Contribution

The paper proposes cyclical pruning, a simple and effective strategy allowing erroneous weights to recover, improving sparsity and accuracy in neural networks.

Findings

Cyclical pruning outperforms existing algorithms at high sparsity ratios.

The method is easy to tune and integrate into existing pipelines.

Experimental results on various models validate its effectiveness.

Abstract

Current methods for pruning neural network weights iteratively apply magnitude-based pruning on the model weights and re-train the resulting model to recover lost accuracy. In this work, we show that such strategies do not allow for the recovery of erroneously pruned weights. To enable weight recovery, we propose a simple strategy called \textit{cyclical pruning} which requires the pruning schedule to be periodic and allows for weights pruned erroneously in one cycle to recover in subsequent ones. Experimental results on both linear models and large-scale deep neural networks show that cyclical pruning outperforms existing pruning algorithms, especially at high sparsity ratios. Our approach is easy to tune and can be readily incorporated into existing pruning pipelines to boost performance.