ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias

TL;DR

ViTAE introduces a novel vision transformer that incorporates intrinsic local and scale-invariance inductive biases through convolutional modules, enhancing feature learning and performance on vision tasks.

Contribution

The paper proposes ViTAE, a vision transformer that embeds multi-scale local features and scale invariance using convolutional modules, improving robustness and efficiency.

Findings

Outperforms baseline transformers on ImageNet

Achieves better feature representation for multi-scale objects

Demonstrates superior results on downstream vision tasks

Abstract

Transformers have shown great potential in various computer vision tasks owing to their strong capability in modeling long-range dependency using the self-attention mechanism. Nevertheless, vision transformers treat an image as 1D sequence of visual tokens, lacking an intrinsic inductive bias (IB) in modeling local visual structures and dealing with scale variance. Alternatively, they require large-scale training data and longer training schedules to learn the IB implicitly. In this paper, we propose a novel Vision Transformer Advanced by Exploring intrinsic IB from convolutions, ie, ViTAE. Technically, ViTAE has several spatial pyramid reduction modules to downsample and embed the input image into tokens with rich multi-scale context by using multiple convolutions with different dilation rates. In this way, it acquires an intrinsic scale invariance IB and is able to learn robust feature representation for objects at various scales. Moreover, in each transformer layer, ViTAE has a convolution block in parallel to the multi-head self-attention module, whose features are fused and fed into the feed-forward network. Consequently, it has the intrinsic locality IB and is able to learn local features and global dependencies collaboratively. Experiments on ImageNet as well as downstream tasks prove the superiority of ViTAE over the baseline transformer and concurrent works. Source code and pretrained models will be available at GitHub.