Channel Pruning In Quantization-aware Training: An Adaptive Projection-gradient Descent-shrinkage-splitting Method

TL;DR

This paper introduces APGDSSM, a novel method combining adaptive projection, gradient descent, and shrinkage techniques to enable efficient channel pruning within quantization-aware training, achieving extreme compression.

Contribution

It presents a new integrated approach that simultaneously optimizes weights for quantization and sparsity, using innovative penalties and splitting techniques for improved compression.

Findings

Effective channel pruning during QAT demonstrated

Achieves significant model compression without accuracy loss

Stabilizes training with a new transformed l1 penalty

Abstract

We propose an adaptive projection-gradient descent-shrinkage-splitting method (APGDSSM) to integrate penalty based channel pruning into quantization-aware training (QAT). APGDSSM concurrently searches weights in both the quantized subspace and the sparse subspace. APGDSSM uses shrinkage operator and a splitting technique to create sparse weights, as well as the Group Lasso penalty to push the weight sparsity into channel sparsity. In addition, we propose a novel complementary transformed l1 penalty to stabilize the training for extreme compression.