StructADMM: A Systematic, High-Efficiency Framework of Structured Weight Pruning for DNNs

TL;DR

This paper introduces StructADMM, a systematic framework for structured weight pruning in DNNs, achieving high sparsity and GPU acceleration without accuracy loss, surpassing prior methods.

Contribution

Proposes a unified ADMM-based framework for various structured sparsity types, significantly improving pruning rate and GPU speedup while maintaining accuracy.

Findings

Achieves 2.58X and 3.65X speedup on GPUs without accuracy loss.

Reaches up to 8.52X speedup with moderate accuracy loss.

Demonstrates higher performance on ResNet, UCF101, and CIFAR-10.

Abstract

Weight pruning methods of DNNs have been demonstrated to achieve a good model pruning rate without loss of accuracy, thereby alleviating the significant computation/storage requirements of large-scale DNNs. Structured weight pruning methods have been proposed to overcome the limitation of irregular network structure and demonstrated actual GPU acceleration. However, in prior work the pruning rate (degree of sparsity) and GPU acceleration are limited (to less than 50%) when accuracy needs to be maintained. In this work,we overcome these limitations by proposing a unified, systematic framework of structured weight pruning for DNNs. It is a framework that can be used to induce different types of structured sparsity, such as filter-wise, channel-wise, and shape-wise sparsity, as well non-structured sparsity. The proposed framework incorporates stochastic gradient descent with ADMM, and can be understood as a dynamic regularization method in which the regularization target is analytically updated in each iteration. Without loss of accuracy on the AlexNet model, we achieve 2.58X and 3.65X average measured speedup on two GPUs, clearly outperforming the prior work. The average speedups reach 3.15X and 8.52X when allowing a moderate ac-curacy loss of 2%. In this case the model compression for convolutional layers is 15.0X, corresponding to 11.93X measured CPU speedup. Our experiments on ResNet model and on other data sets like UCF101 and CIFAR-10 demonstrate the consistently higher performance of our framework.