ViT Registers and Fractal ViT

TL;DR

This paper introduces fractal ViT, a variant of Vision Transformer that uses attention masks with summary tokens to break permutation invariance, but finds it does not outperform ViT with registers, indicating domain-specific effects.

Contribution

The paper proposes fractal ViT, a new model that incorporates attention masks and summary tokens to explore permutation invariance in vision transformers.

Findings

Fractal ViT does not outperform ViT with registers.

Permutation invariance breaking does not universally improve ViT performance.

Findings may be specific to scale, domain, or application.

Abstract

Drawing inspiration from recent findings including surprisingly decent performance of transformers without positional encoding (NoPE) in the domain of language models and how registers (additional throwaway tokens not tied to input) may improve the performance of large vision transformers (ViTs), we invent and test a variant of ViT called fractal ViT that breaks permutation invariance among the tokens by applying an attention mask between the regular tokens and ``summary tokens'' similar to registers, in isolation or in combination with various positional encodings. These models do not improve upon ViT with registers, highlighting the fact that these findings may be scale, domain, or application-specific.