Diversity-Guided MLP Reduction for Efficient Large Vision Transformers

TL;DR

This paper introduces a method to significantly reduce parameters and computational costs of large vision transformers by pruning redundant neurons in MLP modules while maintaining performance, using a diversity-guided approach.

Contribution

We propose a novel Diversity-Guided MLP Reduction (DGMR) method that effectively prunes MLP neurons in vision transformers with minimal performance loss, leveraging weight diversity preservation.

Findings

Achieves over 57% reduction in parameters and FLOPs with negligible performance loss.

Maintains performance with only 0.06% of LAION-2B data for training.

Reduces parameters by 71.5% on EVA-CLIP-E without accuracy degradation.

Abstract

Transformer models achieve excellent scaling property, where the performance is improved with the increment of model capacity. However, large-scale model parameters lead to an unaffordable cost of computing and memory. We analyze popular transformer architectures and find that multilayer perceptron (MLP) modules take up the majority of model parameters. To this end, we focus on the recoverability of the compressed models and propose a Diversity-Guided MLP Reduction (DGMR) method to significantly reduce the parameters of large vision transformers with only negligible performance degradation. Specifically, we conduct a Gram-Schmidt weight pruning strategy to eliminate redundant neurons of MLP hidden layer, while preserving weight diversity for better performance recover during distillation. Compared to the model trained from scratch, our pruned model only requires 0.06\% data of LAION-2B (for the training of large vision transformers) without labels (ImageNet-1K) to recover the original performance. Experimental results on several state-of-the-art large vision transformers demonstrate that our method achieves a more than 57.0\% parameter and FLOPs reduction in a near lossless manner. Notably, for EVA-CLIP-E (4.4B), our method accomplishes a 71.5\% parameter and FLOPs reduction without performance degradation. The source code and trained weights are available at https://github.com/visresearch/DGMR.