Learning to Adapt to Position Bias in Vision Transformer Classifiers

TL;DR

This paper investigates how position bias affects Vision Transformer classifiers and introduces methods to measure and adapt position embeddings, improving classification performance by unlearning unnecessary positional information.

Contribution

It proposes Position-SHAP for measuring position bias and Auto-PE for adaptive position embedding, enabling better handling of position information in datasets.

Findings

Position bias varies across datasets.

Auto-PE improves classification accuracy.

Unlearning position info enhances model robustness.

Abstract

How discriminative position information is for image classification depends on the data. On the one hand, the camera position is arbitrary and objects can appear anywhere in the image, arguing for translation invariance. At the same time, position information is key for exploiting capture/center bias, and scene layout, e.g.: the sky is up. We show that position bias, the level to which a dataset is more easily solved when positional information on input features is used, plays a crucial role in the performance of Vision Transformers image classifiers. To investigate, we propose Position-SHAP, a direct measure of position bias by extending SHAP to work with position embeddings. We show various levels of position bias in different datasets, and find that the optimal choice of position embedding depends on the position bias apparent in the dataset. We therefore propose Auto-PE, a single-parameter position embedding extension, which allows the position embedding to modulate its norm, enabling the unlearning of position information. Auto-PE combines with existing PEs to match or improve accuracy on classification datasets.