Systematic Weight Pruning of DNNs using Alternating Direction Method of Multipliers

TL;DR

This paper introduces a systematic weight pruning method for deep neural networks using ADMM, achieving higher compression ratios and faster convergence while maintaining accuracy.

Contribution

It formulates weight pruning as a constrained nonconvex optimization problem and applies ADMM for efficient, systematic pruning with superior results.

Findings

Higher compression ratios than prior methods

Faster convergence rates in pruning process

Maintained test accuracy after pruning

Abstract

We present a systematic weight pruning framework of deep neural networks (DNNs) using the alternating direction method of multipliers (ADMM). We first formulate the weight pruning problem of DNNs as a constrained nonconvex optimization problem, and then adopt the ADMM framework for systematic weight pruning. We show that ADMM is highly suitable for weight pruning due to the computational efficiency it offers. We achieve a much higher compression ratio compared with prior work while maintaining the same test accuracy, together with a faster convergence rate. Our models are released at https://github.com/KaiqiZhang/admm-pruning