Vision-TTT: Efficient and Expressive Visual Representation Learning with Test-Time Training

TL;DR

Vision-TTT introduces a novel self-supervised learning method for vision transformers that significantly improves efficiency and accuracy by modeling 2D visual correlations with reduced computational complexity.

Contribution

It proposes Vision-TTT, a new linear-time sequence modeling approach that extends test-time training to vision, enhancing scalability and performance of vision transformers.

Findings

Achieves over 77% Top-1 accuracy on ImageNet with ViT models.

Reduces FLOPs by 79.4% and runs 4.72 times faster at high resolution.

Outperforms baseline models on downstream tasks.

Abstract

Learning efficient and expressive visual representation has long been the pursuit of computer vision research. While Vision Transformers (ViTs) gradually replace traditional Convolutional Neural Networks (CNNs) as more scalable vision learners, their applications are plagued by the quadratic complexity of the self-attention mechanism. To address the challenge, we introduce a new linear-time sequence modeling method Test-Time Training (TTT) into vision and propose Vision-TTT, which treats visual sequences as datasets and compresses the visual token sequences in a novel self-supervised learning manner. By incorporating the dual-dataset strategy and Conv2d-based dataset preprocessing, Vision-TTT effectively extends vanilla TTT to model 2D visual correlations with global receptive fields. Extensive experiments show that \texttt{Vittt-T/S/B} achieve $77.7\%,81.8\%,82.7\%$ Top-1 accuracy on ImageNet classification and also greatly outperform their counterparts on downstream tasks. At $1280\times1280$ resolution, \texttt{Vittt-T} reduces FLOPs by $79.4\%$ and runs $4.72\times$ faster with $88.9\%$ less memory than DeiT-T. These results demonstrate the expressiveness and efficiency of Vision-TTT as a strong candidate for the next-generation generic visual backbone.