How Much Information Can a Vision Token Hold? A Scaling Law for Recognition Limits in VLMs

TL;DR

This paper investigates the information capacity of vision tokens in vision-language models, revealing a phase-transition behavior and proposing a scaling law that guides efficient visual context compression.

Contribution

It introduces a probabilistic scaling law for vision tokens, characterizing recognition limits and informing model optimization strategies.

Findings

Identification of three recognition regimes: Stable, Instability, and Collapse.

Formulation of a universal scaling law linking visual density and token load.

Empirical validation across multiple vision-language models.

Abstract

Recent vision-centric approaches have made significant strides in long-context modeling. Represented by DeepSeek-OCR, these models encode rendered text into continuous vision tokens, achieving high compression rates without sacrificing recognition precision. However, viewing the vision encoder as a lossy channel with finite representational capacity raises a fundamental question: what is the information upper bound of visual tokens? To investigate this limit, we conduct controlled stress tests by progressively increasing the information quantity (character count) within an image. We observe a distinct phase-transition phenomenon characterized by three regimes: a near-perfect Stable Phase, an Instability Phase marked by increased error variance, and a total Collapse Phase. We analyze the mechanical origins of these transitions and identify key factors. Furthermore, we formulate a probabilistic scaling law that unifies average vision token load and visual density into a latent difficulty metric. Extensive experiments across various Vision-Language Models demonstrate the universality of this scaling law, providing critical empirical guidance for optimizing the efficiency-accuracy trade-off in visual context compression.