SPoT: Subpixel Placement of Tokens in Vision Transformers

TL;DR

SPoT introduces a novel subpixel token placement method for Vision Transformers, enabling continuous positioning within images, which enhances sparsity exploitation, reduces token count, and improves model efficiency and interpretability.

Contribution

The paper proposes SPoT, a new tokenization strategy that allows continuous token placement, overcoming grid limitations and unlocking performance gains in Vision Transformers.

Findings

Significant performance improvements with ideal subpixel token positioning.

Reduced number of tokens needed for accurate predictions.

Redefines sparsity as an advantage in ViT architectures.

Abstract

Vision Transformers naturally accommodate sparsity, yet standard tokenization methods confine features to discrete patch grids. This constraint prevents models from fully exploiting sparse regimes, forcing awkward compromises. We propose Subpixel Placement of Tokens (SPoT), a novel tokenization strategy that positions tokens continuously within images, effectively sidestepping grid-based limitations. With our proposed oracle-guided search, we uncover substantial performance gains achievable with ideal subpixel token positioning, drastically reducing the number of tokens necessary for accurate predictions during inference. SPoT provides a new direction for flexible, efficient, and interpretable ViT architectures, redefining sparsity as a strategic advantage rather than an imposed limitation.