Isomorphic Pruning for Vision Models

TL;DR

Isomorphic Pruning introduces a method for more reliable structured pruning of vision models by comparing isomorphic sub-structures separately, leading to improved accuracy and efficiency across CNNs and Transformers.

Contribution

The paper proposes Isomorphic Pruning, a novel approach that compares importance of sub-structures separately to enhance pruning effectiveness in diverse vision models.

Findings

Outperforms existing pruning baselines on ImageNet-1K.

Improves DeiT-Tiny accuracy from 74.52% to 77.50%.

Slightly increases ConvNext-Tiny accuracy from 82.06% to 82.18%.

Abstract

Structured pruning reduces the computational overhead of deep neural networks by removing redundant sub-structures. However, assessing the relative importance of different sub-structures remains a significant challenge, particularly in advanced vision models featuring novel mechanisms and architectures like self-attention, depth-wise convolutions, or residual connections. These heterogeneous substructures usually exhibit diverged parameter scales, weight distributions, and computational topology, introducing considerable difficulty to importance comparison. To overcome this, we present Isomorphic Pruning, a simple approach that demonstrates effectiveness across a range of network architectures such as Vision Transformers and CNNs, and delivers competitive performance across different model sizes. Isomorphic Pruning originates from an observation that, when evaluated under a pre-defined importance criterion, heterogeneous sub-structures demonstrate significant divergence in their importance distribution, as opposed to isomorphic structures that present similar importance patterns. This inspires us to perform isolated ranking and comparison on different types of sub-structures for more reliable pruning. Our empirical results on ImageNet-1K demonstrate that Isomorphic Pruning surpasses several pruning baselines dedicatedly designed for Transformers or CNNs. For instance, we improve the accuracy of DeiT-Tiny from 74.52% to 77.50% by pruning an off-the-shelf DeiT-Base model. And for ConvNext-Tiny, we enhanced performance from 82.06% to 82.18%, while reducing the number of parameters and memory usage. Code is available at \url{https://github.com/VainF/Isomorphic-Pruning}.