Spatial Entropy as an Inductive Bias for Vision Transformers

TL;DR

This paper introduces a novel regularization method for Vision Transformers that uses spatial entropy as an auxiliary self-supervised task, encouraging semantic segmentation structures and improving accuracy especially with limited training data.

Contribution

It proposes a new spatial entropy-based regularization technique that enhances Vision Transformers without altering their architecture, leveraging self-supervised learning to induce a local spatial bias.

Findings

Regularization improves Vision Transformer accuracy with small-medium datasets.

Method matches or exceeds performance of architecture-based local bias methods.

Spatial entropy regularization enhances semantic clustering in attention maps.

Abstract

Recent work on Vision Transformers (VTs) showed that introducing a local inductive bias in the VT architecture helps reducing the number of samples necessary for training. However, the architecture modifications lead to a loss of generality of the Transformer backbone, partially contradicting the push towards the development of uniform architectures, shared, e.g., by both the Computer Vision and the Natural Language Processing areas. In this work, we propose a different and complementary direction, in which a local bias is introduced using an auxiliary self-supervised task, performed jointly with standard supervised training. Specifically, we exploit the observation that the attention maps of VTs, when trained with self-supervision, can contain a semantic segmentation structure which does not spontaneously emerge when training is supervised. Thus, we explicitly encourage the emergence of this spatial clustering as a form of training regularization. In more detail, we exploit the assumption that, in a given image, objects usually correspond to few connected regions, and we propose a spatial formulation of the information entropy to quantify this object-based inductive bias. By minimizing the proposed spatial entropy, we include an additional self-supervised signal during training. Using extensive experiments, we show that the proposed regularization leads to equivalent or better results than other VT proposals which include a local bias by changing the basic Transformer architecture, and it can drastically boost the VT final accuracy when using small-medium training sets. The code is available at https://github.com/helia95/SAR.