Adaptively Bypassing Vision Transformer Blocks for Efficient Visual Tracking

TL;DR

ABTrack introduces an adaptive framework that selectively bypasses transformer blocks in visual tracking, significantly improving speed while maintaining state-of-the-art accuracy by dynamically simplifying the model based on scene and target characteristics.

Contribution

The paper proposes a novel adaptive bypass mechanism and a ViT pruning method to enhance the efficiency of transformer-based visual trackers without sacrificing performance.

Findings

Achieves state-of-the-art tracking performance on multiple benchmarks.

Significantly reduces inference time compared to existing methods.

Demonstrates the effectiveness of adaptive bypassing and pruning in real-world scenarios.

Abstract

Empowered by transformer-based models, visual tracking has advanced significantly. However, the slow speed of current trackers limits their applicability on devices with constrained computational resources. To address this challenge, we introduce ABTrack, an adaptive computation framework that adaptively bypassing transformer blocks for efficient visual tracking. The rationale behind ABTrack is rooted in the observation that semantic features or relations do not uniformly impact the tracking task across all abstraction levels. Instead, this impact varies based on the characteristics of the target and the scene it occupies. Consequently, disregarding insignificant semantic features or relations at certain abstraction levels may not significantly affect the tracking accuracy. We propose a Bypass Decision Module (BDM) to determine if a transformer block should be bypassed, which adaptively simplifies the architecture of ViTs and thus speeds up the inference process. To counteract the time cost incurred by the BDMs and further enhance the efficiency of ViTs, we introduce a novel ViT pruning method to reduce the dimension of the latent representation of tokens in each transformer block. Extensive experiments on multiple tracking benchmarks validate the effectiveness and generality of the proposed method and show that it achieves state-of-the-art performance. Code is released at: https://github.com/xyyang317/ABTrack.