Variance-Based Pruning for Accelerating and Compressing Trained Networks

TL;DR

This paper introduces Variance-Based Pruning, a one-shot structured pruning method that efficiently compresses trained networks with minimal fine-tuning, maintaining high accuracy and significantly reducing computational costs.

Contribution

It proposes a novel pruning technique that uses activation statistics to select neurons, preserving performance with little retraining, and demonstrates its effectiveness on ImageNet-1k.

Findings

Retains over 70% of DeiT-Base performance immediately after pruning.

Requires only 10 epochs of fine-tuning to recover 99% of original accuracy.

Reduces MACs by 35% and model size by 36%, speeding up inference by 1.44x.

Abstract

Increasingly expensive training of ever larger models such as Vision Transfomers motivate reusing the vast library of already trained state-of-the-art networks. However, their latency, high computational costs and memory demands pose significant challenges for deployment, especially on resource-constrained hardware. While structured pruning methods can reduce these factors, they often require costly retraining, sometimes for up to hundreds of epochs, or even training from scratch to recover the lost accuracy resulting from the structural modifications. Maintaining the provided performance of trained models after structured pruning and thereby avoiding extensive retraining remains a challenge. To solve this, we introduce Variance-Based Pruning, a simple and structured one-shot pruning technique for efficiently compressing networks, with minimal finetuning. Our approach first gathers activation statistics, which are used to select neurons for pruning. Simultaneously the mean activations are integrated back into the model to preserve a high degree of performance. On ImageNet-1k recognition tasks, we demonstrate that directly after pruning DeiT-Base retains over 70% of its original performance and requires only 10 epochs of fine-tuning to regain 99% of the original accuracy while simultaneously reducing MACs by 35% and model size by 36%, thus speeding up the model by 1.44x. The code is available at: https://github.com/boschresearch/variance-based-pruning