UniPTS: A Unified Framework for Proficient Post-Training Sparsity

TL;DR

UniPTS is a comprehensive framework that significantly improves post-training sparsity performance by integrating a decay-based sparsity objective, an optimal sparsity search algorithm, and dynamic sparse training, outperforming existing methods on benchmarks.

Contribution

The paper introduces UniPTS, a novel unified framework that enhances post-training sparsity by addressing key factors affecting performance, with innovative algorithms and training strategies.

Findings

Substantially improves sparsity performance on benchmarks.

Achieves 68.6% accuracy at 90% sparsity on ResNet-50/ImageNet.

Outperforms existing PTS methods significantly.

Abstract

Post-training Sparsity (PTS) is a recently emerged avenue that chases efficient network sparsity with limited data in need. Existing PTS methods, however, undergo significant performance degradation compared with traditional methods that retrain the sparse networks via the whole dataset, especially at high sparsity ratios. In this paper, we attempt to reconcile this disparity by transposing three cardinal factors that profoundly alter the performance of conventional sparsity into the context of PTS. Our endeavors particularly comprise (1) A base-decayed sparsity objective that promotes efficient knowledge transferring from dense network to the sparse counterpart. (2) A reducing-regrowing search algorithm designed to ascertain the optimal sparsity distribution while circumventing overfitting to the small calibration set in PTS. (3) The employment of dynamic sparse training predicated on the preceding aspects, aimed at comprehensively optimizing the sparsity structure while ensuring training stability. Our proposed framework, termed UniPTS, is validated to be much superior to existing PTS methods across extensive benchmarks. As an illustration, it amplifies the performance of POT, a recently proposed recipe, from 3.9% to 68.6% when pruning ResNet-50 at 90% sparsity ratio on ImageNet. We release the code of our paper at https://github.com/xjjxmu/UniPTS.