Inducing Spatial Locality in Vision Transformers through the Training Protocol

TL;DR

This paper demonstrates that a specific training protocol, especially the use of CutMix augmentation, can induce spatial locality in Vision Transformers trained from scratch, without large-scale pretraining.

Contribution

It shows that training protocols, notably CutMix, can promote local attention in early layers of Vision Transformers without pretraining.

Findings

CutMix significantly reduces Mean Attention Distance (MAD) indicating increased locality.

AutoAugment and Label Smoothing do not independently affect attention locality.

Training with CutMix leads to more concentrated and local attention in early layers.

Abstract

We investigate whether the training protocol can induce spatial locality in the early layers of a Vision Transformer (ViT) trained from scratch, without large-scale pretraining. Keeping the architecture and optimization procedure fixed, we compare a Baseline protocol with a Modern protocol (AutoAugment/ColorJitter, CutMix, and Label Smoothing) on CIFAR-10, CIFAR-100, and Tiny-ImageNet, characterizing each attention head via Mean Attention Distance (MAD) and normalized entropy. Across all three datasets, the Modern protocol produces more local and more concentrated attention in early layers; on CIFAR-100, the minimum MAD drops from 0.316 (Baseline) to 0.008 (Modern). To identify the source of this effect, we conduct an ablation study on CIFAR-100 by adding or removing each component individually. The results identify CutMix as the determining component within our experiments: all conditions with CutMix exhibit MAD 0.024, while all conditions without CutMix remain at MAD 0.210. AutoAugment and Label Smoothing show no independent effect on locality. Taken together, these findings suggest that the pressure to classify from partial image regions, induced by CutMix, can promote the emergence of local attention in Vision Transformers.