Revisiting [CLS] and Patch Token Interaction in Vision Transformers

TL;DR

This paper investigates the interaction between class and patch tokens in Vision Transformers, revealing normalization layers' role and proposing specialized processing to improve dense prediction tasks without significant overhead.

Contribution

It introduces targeted processing paths that disentangle class and patch token flows, enhancing dense prediction performance in Vision Transformers.

Findings

Segmentation performance improved by over 2 mIoU points.

Normalization layers cause implicit differentiation between token types.

Proposed modifications add 8% parameters with no extra computational cost.

Abstract

Vision Transformers have emerged as powerful, scalable and versatile representation learners. To capture both global and local features, a learnable [CLS] class token is typically prepended to the input sequence of patch tokens. Despite their distinct nature, both token types are processed identically throughout the model. In this work, we investigate the friction between global and local feature learning under different pre-training strategies by analyzing the interactions between class and patch tokens. Our analysis reveals that standard normalization layers introduce an implicit differentiation between these token types. Building on this insight, we propose specialized processing paths that selectively disentangle the computational flow of class and patch tokens, particularly within normalization layers and early query-key-value projections. This targeted specialization leads to significantly improved patch representation quality for dense prediction tasks. Our experiments demonstrate segmentation performance gains of over 2 mIoU points on standard benchmarks, while maintaining strong classification accuracy. The proposed modifications introduce only an 8% increase in parameters, with no additional computational overhead. Through comprehensive ablations, we provide insights into which architectural components benefit most from specialization and how our approach generalizes across model scales and learning frameworks.