ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgetting

TL;DR

ResRep introduces a lossless CNN pruning method inspired by neurobiology, which decouples remembering and forgetting to achieve high compression without accuracy loss.

Contribution

It proposes a novel re-parameterization approach that enables lossless channel pruning by separating and merging remembering and forgetting components.

Findings

Achieves 45% FLOPs reduction on ResNet-50 with no accuracy drop.

First to demonstrate lossless pruning at such a high compression ratio.

Utilizes a novel update rule with penalty gradients for structured sparsity.

Abstract

We propose ResRep, a novel method for lossless channel pruning (a.k.a. filter pruning), which slims down a CNN by reducing the width (number of output channels) of convolutional layers. Inspired by the neurobiology research about the independence of remembering and forgetting, we propose to re-parameterize a CNN into the remembering parts and forgetting parts, where the former learn to maintain the performance and the latter learn to prune. Via training with regular SGD on the former but a novel update rule with penalty gradients on the latter, we realize structured sparsity. Then we equivalently merge the remembering and forgetting parts into the original architecture with narrower layers. In this sense, ResRep can be viewed as a successful application of Structural Re-parameterization. Such a methodology distinguishes ResRep from the traditional learning-based pruning paradigm that applies a penalty on parameters to produce sparsity, which may suppress the parameters essential for the remembering. ResRep slims down a standard ResNet-50 with 76.15% accuracy on ImageNet to a narrower one with only 45% FLOPs and no accuracy drop, which is the first to achieve lossless pruning with such a high compression ratio. The code and models are at https://github.com/DingXiaoH/ResRep.