Searching Intrinsic Dimensions of Vision Transformers

TL;DR

This paper introduces SiDT, a pruning method for vision transformers that reduces computational costs while maintaining or improving performance on complex vision tasks like object detection.

Contribution

We propose SiDT, a novel dimension search-based pruning method for vision transformers applied to complex tasks beyond image classification.

Findings

Pruned models with 20-40% dimensions outperform unpruned models.

SiDT achieves comparable or better accuracy with reduced complexity.

The method outperforms previous pruning techniques in efficiency and effectiveness.

Abstract

It has been shown by many researchers that transformers perform as well as convolutional neural networks in many computer vision tasks. Meanwhile, the large computational costs of its attention module hinder further studies and applications on edge devices. Some pruning methods have been developed to construct efficient vision transformers, but most of them have considered image classification tasks only. Inspired by these results, we propose SiDT, a method for pruning vision transformer backbones on more complicated vision tasks like object detection, based on the search of transformer dimensions. Experiments on CIFAR-100 and COCO datasets show that the backbones with 20\% or 40\% dimensions/parameters pruned can have similar or even better performance than the unpruned models. Moreover, we have also provided the complexity analysis and comparisons with the previous pruning methods.