LGViT: Dynamic Early Exiting for Accelerating Vision Transformer

TL;DR

This paper introduces LGViT, a novel early exiting framework for vision transformers that balances efficiency and accuracy, achieving significant speed-up with minimal performance loss on multiple datasets.

Contribution

The paper proposes a new early exiting method for ViTs with heterogeneous heads and a two-stage training scheme, improving inference speed while maintaining accuracy.

Findings

LGViT achieves approximately 1.8× speed-up.

Extensive experiments validate LGViT's effectiveness across multiple ViT backbones.

The method maintains competitive performance with reduced inference time.

Abstract

Recently, the efficient deployment and acceleration of powerful vision transformers (ViTs) on resource-limited edge devices for providing multimedia services have become attractive tasks. Although early exiting is a feasible solution for accelerating inference, most works focus on convolutional neural networks (CNNs) and transformer models in natural language processing (NLP).Moreover, the direct application of early exiting methods to ViTs may result in substantial performance degradation. To tackle this challenge, we systematically investigate the efficacy of early exiting in ViTs and point out that the insufficient feature representations in shallow internal classifiers and the limited ability to capture target semantic information in deep internal classifiers restrict the performance of these methods. We then propose an early exiting framework for general ViTs termed LGViT, which incorporates heterogeneous exiting heads, namely, local perception head and global aggregation head, to achieve an efficiency-accuracy trade-off. In particular, we develop a novel two-stage training scheme, including end-to-end training and self-distillation with the backbone frozen to generate early exiting ViTs, which facilitates the fusion of global and local information extracted by the two types of heads. We conduct extensive experiments using three popular ViT backbones on three vision datasets. Results demonstrate that our LGViT can achieve competitive performance with approximately 1.8 $\times$ speed-up.