Soft Tail-dropping for Adaptive Visual Tokenization

TL;DR

This paper introduces STAT, a novel adaptive visual tokenizer that dynamically adjusts token count based on image complexity, improving autoregressive visual generation quality and scalability.

Contribution

The paper proposes a new 1D discrete visual tokenizer, STAT, that adaptively determines token length based on image complexity, enhancing AR model performance.

Findings

STAT achieves competitive or superior generation quality on ImageNet-1k.

The method exhibits favorable scaling behavior in autoregressive visual generation.

Adaptive tokenization improves compatibility with causal AR models.

Abstract

We present Soft Tail-dropping Adaptive Tokenizer (STAT), a 1D discrete visual tokenizer that adaptively chooses the number of output tokens per image according to its structural complexity and level of detail. STAT encodes an image into a sequence of discrete codes together with per-token keep probabilities. Beyond standard autoencoder objectives, we regularize these keep probabilities to be monotonically decreasing along the sequence and explicitly align their distribution with an image-level complexity measure. As a result, STAT produces length-adaptive 1D visual tokens that are naturally compatible with causal 1D autoregressive (AR) visual generative models. On ImageNet-1k, equipping vanilla causal AR models with STAT yields competitive or superior visual generation quality compared to other probabilistic model families, while also exhibiting favorable scaling behavior that has been elusive in prior vanilla AR visual generation attempts.