ViT-P: Rethinking Data-efficient Vision Transformers from Locality

TL;DR

This paper introduces ViT-P, a data-efficient vision transformer that uses multi-focal attention bias to improve training on small datasets by constraining the self-attention to local receptive fields, achieving state-of-the-art results.

Contribution

The paper proposes a novel multi-focal attention bias mechanism that adaptively constrains the receptive field in vision transformers, enhancing data efficiency and performance.

Findings

ViT-P achieves 83.16% accuracy on Cifar100 from scratch.

Proper receptive field constraints reduce training data requirements.

Method maintains accuracy on large datasets like ImageNet.

Abstract

Recent advances of Transformers have brought new trust to computer vision tasks. However, on small dataset, Transformers is hard to train and has lower performance than convolutional neural networks. We make vision transformers as data-efficient as convolutional neural networks by introducing multi-focal attention bias. Inspired by the attention distance in a well-trained ViT, we constrain the self-attention of ViT to have multi-scale localized receptive field. The size of receptive field is adaptable during training so that optimal configuration can be learned. We provide empirical evidence that proper constrain of receptive field can reduce the amount of training data for vision transformers. On Cifar100, our ViT-P Base model achieves the state-of-the-art accuracy (83.16%) trained from scratch. We also perform analysis on ImageNet to show our method does not lose accuracy on large data sets.