A Unified Framework of DNN Weight Pruning and Weight Clustering/Quantization Using ADMM

TL;DR

This paper introduces a unified ADMM-based framework for DNN weight pruning and clustering/quantization, significantly improving model compression while maintaining accuracy.

Contribution

It develops a systematic, unified optimization framework combining weight pruning and clustering/quantization using ADMM, enabling joint solutions and enhanced compression.

Findings

Achieves up to 167x weight reduction in LeNet-5 without accuracy loss.

Reaches 1,910x storage reduction in LeNet-5 with combined pruning and quantization.

Demonstrates significant improvements over existing methods in model compression.

Abstract

Many model compression techniques of Deep Neural Networks (DNNs) have been investigated, including weight pruning, weight clustering and quantization, etc. Weight pruning leverages the redundancy in the number of weights in DNNs, while weight clustering/quantization leverages the redundancy in the number of bit representations of weights. They can be effectively combined in order to exploit the maximum degree of redundancy. However, there lacks a systematic investigation in literature towards this direction. In this paper, we fill this void and develop a unified, systematic framework of DNN weight pruning and clustering/quantization using Alternating Direction Method of Multipliers (ADMM), a powerful technique in optimization theory to deal with non-convex optimization problems. Both DNN weight pruning and clustering/quantization, as well as their combinations, can be solved in a unified manner. For further performance improvement in this framework, we adopt multiple techniques including iterative weight quantization and retraining, joint weight clustering training and centroid updating, weight clustering retraining, etc. The proposed framework achieves significant improvements both in individual weight pruning and clustering/quantization problems, as well as their combinations. For weight pruning alone, we achieve 167x weight reduction in LeNet-5, 24.7x in AlexNet, and 23.4x in VGGNet, without any accuracy loss. For the combination of DNN weight pruning and clustering/quantization, we achieve 1,910x and 210x storage reduction of weight data on LeNet-5 and AlexNet, respectively, without accuracy loss. Our codes and models are released at the link http://bit.ly/2D3F0np