Boosting Pruned Networks with Linear Over-parameterization

TL;DR

This paper introduces a method to improve accuracy of pruned neural networks by linearly over-parameterizing layers during fine-tuning and then re-parameterizing them back, enhanced with similarity-preserving knowledge distillation.

Contribution

The paper proposes a novel linear over-parameterization technique combined with knowledge distillation to better restore accuracy in pruned networks after compression.

Findings

Significantly outperforms vanilla fine-tuning on CIFAR-10 and ImageNet.

Effective especially at large pruning ratios.

Enables more accurate fine-tuning of highly compressed networks.

Abstract

Structured pruning compresses neural networks by reducing channels (filters) for fast inference and low footprint at run-time. To restore accuracy after pruning, fine-tuning is usually applied to pruned networks. However, too few remaining parameters in pruned networks inevitably bring a great challenge to fine-tuning to restore accuracy. To address this challenge, we propose a novel method that first linearly over-parameterizes the compact layers in pruned networks to enlarge the number of fine-tuning parameters and then re-parameterizes them to the original layers after fine-tuning. Specifically, we equivalently expand the convolution/linear layer with several consecutive convolution/linear layers that do not alter the current output feature maps. Furthermore, we utilize similarity-preserving knowledge distillation that encourages the over-parameterized block to learn the immediate data-to-data similarities of the corresponding dense layer to maintain its feature learning ability. The proposed method is comprehensively evaluated on CIFAR-10 and ImageNet which significantly outperforms the vanilla fine-tuning strategy, especially for large pruning ratio.