Feather: An Elegant Solution to Effective DNN Sparsification

TL;DR

Feather is a novel sparse training module that efficiently prunes neural networks during training, achieving state-of-the-art accuracy on ImageNet with ResNet-50 and demonstrating broad applicability across architectures.

Contribution

Introducing Feather, a new sparsification method utilizing the Straight-Through Estimator, a thresholding operator, and gradient scaling for effective neural network pruning during training.

Findings

Achieves state-of-the-art Top-1 accuracy on ImageNet with ResNet-50.

Demonstrates robustness and adaptability across various architectures.

Outperforms existing pruning methods, including complex ones.

Abstract

Neural Network pruning is an increasingly popular way for producing compact and efficient models, suitable for resource-limited environments, while preserving high performance. While the pruning can be performed using a multi-cycle training and fine-tuning process, the recent trend is to encompass the sparsification process during the standard course of training. To this end, we introduce Feather, an efficient sparse training module utilizing the powerful Straight-Through Estimator as its core, coupled with a new thresholding operator and a gradient scaling technique, enabling robust, out-of-the-box sparsification performance. Feather's effectiveness and adaptability is demonstrated using various architectures on the CIFAR dataset, while on ImageNet it achieves state-of-the-art Top-1 validation accuracy using the ResNet-50 architecture, surpassing existing methods, including more complex and computationally heavy ones, by a considerable margin. Code is publicly available at https://github.com/athglentis/feather .