What DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformers

TL;DR

This paper investigates positional biases in Vision Transformers like DINOv2 and demonstrates that using ALiBi relative positional encoding reduces these biases, improving zero-shot adaptation in microstructure image analysis.

Contribution

The study shows that ALiBi encoding effectively reduces positional bias in ViTs without losing semantic understanding, enhancing their application in scientific imaging.

Findings

ALiBi reduces positional bias in ViTs.

Unbiased features improve microscopy image segmentation.

Models retain semantic richness after ALiBi finetuning.

Abstract

Vision transformers (ViTs) - especially feature foundation models like DINOv2 - learn rich representations useful for many downstream tasks. However, architectural choices (such as positional encoding) can lead to these models displaying positional biases and artefacts independent of semantic content. This makes zero-shot adaption difficult in fields like material science, where images are often cross-sections of homogeneous microstructure (i.e. having no preferred direction). In this work, we investigate the positional bias in ViTs via linear probing, finding it present across a range of objectives and positional encodings, and subsequently reduce it by finetuning models to use ALiBi relative positional encoding. We demonstrate that these models retain desirable general semantics and their unbiased features can be used successfully in trainable segmentation of complex microscopy images.