Fast and Controllable Post-training Sparsity: Learning Optimal Sparsity Allocation with Global Constraint in Minutes

TL;DR

This paper introduces FCPTS, a fast and controllable post-training sparsity method that efficiently allocates sparsity across neural network layers with guaranteed global sparsity, significantly improving accuracy over previous approaches.

Contribution

The paper proposes a novel differentiable and controllable framework for post-training sparsity allocation that converges quickly and surpasses state-of-the-art accuracy improvements.

Findings

Achieves over 30% accuracy improvement on ResNet-50 with 80% sparsity.

Learns optimal layer-wise sparsity in minutes with guaranteed global sparsity.

Outperforms existing post-training sparsity methods by a large margin.

Abstract

Neural network sparsity has attracted many research interests due to its similarity to biological schemes and high energy efficiency. However, existing methods depend on long-time training or fine-tuning, which prevents large-scale applications. Recently, some works focusing on post-training sparsity (PTS) have emerged. They get rid of the high training cost but usually suffer from distinct accuracy degradation due to neglect of the reasonable sparsity rate at each layer. Previous methods for finding sparsity rates mainly focus on the training-aware scenario, which usually fails to converge stably under the PTS setting with limited data and much less training cost. In this paper, we propose a fast and controllable post-training sparsity (FCPTS) framework. By incorporating a differentiable bridge function and a controllable optimization objective, our method allows for rapid and accurate sparsity allocation learning in minutes, with the added assurance of convergence to a predetermined global sparsity rate. Equipped with these techniques, we can surpass the state-of-the-art methods by a large margin, e.g., over 30\% improvement for ResNet-50 on ImageNet under the sparsity rate of 80\%. Our plug-and-play code and supplementary materials are open-sourced at https://github.com/ModelTC/FCPTS.