Efficient Joint Optimization of Layer-Adaptive Weight Pruning in Deep Neural Networks

TL;DR

This paper introduces a fast, layer-adaptive weight pruning method for deep neural networks that optimizes output distortion collectively across layers, leading to significant accuracy improvements on benchmark datasets.

Contribution

The paper presents a novel, efficient pruning scheme that considers all layers simultaneously and leverages an additivity property to solve the optimization with dynamic programming.

Findings

Achieves up to 4.7% higher top-1 accuracy on ImageNet.

Outperforms existing pruning methods on CIFAR-10 and ImageNet datasets.

Demonstrates linear time complexity for the pruning optimization.

Abstract

In this paper, we propose a novel layer-adaptive weight-pruning approach for Deep Neural Networks (DNNs) that addresses the challenge of optimizing the output distortion minimization while adhering to a target pruning ratio constraint. Our approach takes into account the collective influence of all layers to design a layer-adaptive pruning scheme. We discover and utilize a very important additivity property of output distortion caused by pruning weights on multiple layers. This property enables us to formulate the pruning as a combinatorial optimization problem and efficiently solve it through dynamic programming. By decomposing the problem into sub-problems, we achieve linear time complexity, making our optimization algorithm fast and feasible to run on CPUs. Our extensive experiments demonstrate the superiority of our approach over existing methods on the ImageNet and CIFAR-10 datasets. On CIFAR-10, our method achieves remarkable improvements, outperforming others by up to 1.0% for ResNet-32, 0.5% for VGG-16, and 0.7% for DenseNet-121 in terms of top-1 accuracy. On ImageNet, we achieve up to 4.7% and 4.6% higher top-1 accuracy compared to other methods for VGG-16 and ResNet-50, respectively. These results highlight the effectiveness and practicality of our approach for enhancing DNN performance through layer-adaptive weight pruning. Code will be available on https://github.com/Akimoto-Cris/RD_VIT_PRUNE.