Effective Model Sparsification by Scheduled Grow-and-Prune Methods

TL;DR

This paper introduces a scheduled grow-and-prune (GaP) method for neural network sparsification that does not require pre-training, achieving high sparsity levels while maintaining or improving model accuracy across various tasks.

Contribution

The proposed GaP methodology enables effective model sparsification without pre-training, outperforming existing methods in accuracy at high sparsity levels.

Findings

Models pruned with GaP match or surpass dense model quality at 80% sparsity.

GaP achieves 77.9% top-1 accuracy on ImageNet with 90% sparsity, surpassing previous SOTA.

The method is effective across multiple tasks including image classification, detection, segmentation, and translation.

Abstract

Deep neural networks (DNNs) are effective in solving many real-world problems. Larger DNN models usually exhibit better quality (e.g., accuracy) but their excessive computation results in long inference time. Model sparsification can reduce the computation and memory cost while maintaining model quality. Most existing sparsification algorithms unidirectionally remove weights, while others randomly or greedily explore a small subset of weights in each layer for pruning. The limitations of these algorithms reduce the level of achievable sparsity. In addition, many algorithms still require pre-trained dense models and thus suffer from large memory footprint. In this paper, we propose a novel scheduled grow-and-prune (GaP) methodology without having to pre-train a dense model. It addresses the shortcomings of the previous works by repeatedly growing a subset of layers to dense and then pruning them back to sparse after some training. Experiments show that the models pruned using the proposed methods match or beat the quality of the highly optimized dense models at 80% sparsity on a variety of tasks, such as image classification, objective detection, 3D object part segmentation, and translation. They also outperform other state-of-the-art (SOTA) methods for model sparsification. As an example, a 90% non-uniform sparse ResNet-50 model obtained via GaP achieves 77.9% top-1 accuracy on ImageNet, improving the previous SOTA results by 1.5%. Code available at: https://github.com/boone891214/GaP.