ViT-5: Vision Transformers for The Mid-2020s

TL;DR

This paper introduces ViT-5, an improved Vision Transformer architecture that incorporates recent advancements, leading to better performance on classification and generative tasks while maintaining simplicity and transferability.

Contribution

ViT-5 systematically modernizes Vision Transformers with component-wise refinements, achieving state-of-the-art results and better generalization for understanding and generation tasks.

Findings

ViT-5-Base achieves 84.2% top-1 accuracy on ImageNet-1k.

ViT-5 improves FID scores in generative modeling tasks.

ViT-5 demonstrates enhanced representation learning and spatial reasoning.

Abstract

This work presents a systematic investigation into modernizing Vision Transformer backbones by leveraging architectural advancements from the past five years. While preserving the canonical Attention-FFN structure, we conduct a component-wise refinement involving normalization, activation functions, positional encoding, gating mechanisms, and learnable tokens. These updates form a new generation of Vision Transformers, which we call ViT-5. Extensive experiments demonstrate that ViT-5 consistently outperforms state-of-the-art plain Vision Transformers across both understanding and generation benchmarks. On ImageNet-1k classification, ViT-5-Base reaches 84.2\% top-1 accuracy under comparable compute, exceeding DeiT-III-Base at 83.8\%. ViT-5 also serves as a stronger backbone for generative modeling: when plugged into an SiT diffusion framework, it achieves 1.84 FID versus 2.06 with a vanilla ViT backbone. Beyond headline metrics, ViT-5 exhibits improved representation learning and favorable spatial reasoning behavior, and transfers reliably across tasks. With a design aligned with contemporary foundation-model practices, ViT-5 offers a simple drop-in upgrade over vanilla ViT for mid-2020s vision backbones.