Vision Transformer Pruning Via Matrix Decomposition

TL;DR

This paper advances Vision Transformer pruning by integrating matrix decomposition techniques, notably Singular Value Decomposition, to effectively reduce model complexity while maintaining accuracy.

Contribution

It introduces and compares multiple matrix decomposition methods for pruning Vision Transformers, selecting SVD as the most effective approach.

Findings

SVD preserves accuracy better than other decompositions.

Matrix decomposition reduces model size and computational demands.

SVD outperforms QR and LU in maintaining original accuracy.

Abstract

This is a further development of Vision Transformer Pruning via matrix decomposition. The purpose of the Vision Transformer Pruning is to prune the dimension of the linear projection of the dataset by learning their associated importance score in order to reduce the storage, run-time memory, and computational demands. In this paper we further reduce dimension and complexity of the linear projection by implementing and comparing several matrix decomposition methods while preserving the generated important features. We end up selected the Singular Value Decomposition as the method to achieve our goal by comparing the original accuracy scores in the original Github repository and the accuracy scores of using those matrix decomposition methods, including Singular Value Decomposition, four versions of QR Decomposition, and LU factorization.