Register and [CLS] tokens yield a decoupling of local and global features in large ViTs

TL;DR

This paper investigates how register and [CLS] tokens in large Vision Transformers affect the relationship between local and global features, revealing a decoupling that impacts interpretability and suggesting ways to improve model transparency.

Contribution

The study uncovers that register and [CLS] tokens cause a decoupling of local and global features in large ViTs, affecting attention map interpretability and proposing insights for more transparent models.

Findings

Register tokens produce cleaner attention maps.

Global information is dominated by register tokens.

[CLS] token causes similar decoupling in models without explicit register tokens.

Abstract

Recent work has shown that the attention maps of the widely popular DINOv2 model exhibit artifacts, which hurt both model interpretability and performance on dense image tasks. These artifacts emerge due to the model repurposing patch tokens with redundant local information for the storage of global image information. To address this problem, additional register tokens have been incorporated in which the model can store such information instead. We carefully examine the influence of these register tokens on the relationship between global and local image features, showing that while register tokens yield cleaner attention maps, these maps do not accurately reflect the integration of local image information in large models. Instead, global information is dominated by information extracted from register tokens, leading to a disconnect between local and global features. Inspired by these findings, we show that the [CLS] token itself leads to a very similar phenomenon in models without explicit register tokens. Our work shows that care must be taken when interpreting attention maps of large ViTs. Further, by clearly attributing the faulty behavior to register and [CLS] tokens, we show a path towards more interpretable vision models.