Fewer Tokens, Greater Scaling: Self-Adaptive Visual Bases for Efficient and Expansive Representation Learning

TL;DR

This paper introduces a novel approach to reduce visual tokens in models by adaptively clustering redundant tokens into orthogonal bases, enabling more efficient and scalable image representation learning.

Contribution

We propose Orthogonal Filtering, a lightweight module that adaptively reduces tokens, and establish a scaling law showing larger models need fewer tokens for semantic representation.

Findings

Larger models require fewer tokens to span visual semantic space.

Orthogonal Filtering effectively clusters redundant tokens into orthogonal bases.

The approach improves efficiency and scalability in vision models.

Abstract

This paper investigates the fundamental relationship between model capacity and the minimal number of visual tokens required to preserve image semantics. Inspired by the Minimum Description Length principle, we reinterpret image tokens as vectors in a visual semantic space and define the intrinsic semantic complexity of an image as the smallest set of basis vectors needed to span this space. Building on this perspective, we propose Orthogonal Filtering, a lightweight module that adaptively clusters redundant tokens into a compact set of orthogonal bases. Through extensive experiments across a range of ViT models, we reveal a consistent token, model scaling law: larger models require significantly fewer tokens to span visual semantic space. Besides, we also contribute a visual long-context dataset.