Improving Flexible Image Tokenizers for Autoregressive Image Generation

TL;DR

This paper introduces ReToK, a novel flexible image tokenizer that enhances autoregressive image generation by alleviating information concentration issues through redundant token padding and hierarchical semantic regularization, leading to improved performance.

Contribution

ReToK is the first flexible tokenizer combining redundant token padding and hierarchical semantic regularization to fully utilize all tokens for better image modeling.

Findings

ReToK outperforms existing tokenizers on ImageNet 256x256.

Redundant token padding improves tail token utilization.

Hierarchical semantic regularization enhances feature alignment.

Abstract

Flexible image tokenizers aim to represent an image using an ordered 1D variable-length token sequence. This flexible tokenization is typically achieved through nested dropout, where a portion of trailing tokens is randomly truncated during training, and the image is reconstructed using the remaining preceding sequence. However, this tail-truncation strategy inherently concentrates the image information in the early tokens, limiting the effectiveness of downstream AutoRegressive (AR) image generation as the token length increases. To overcome these limitations, we propose \textbf{ReToK}, a flexible tokenizer with \underline{Re}dundant \underline{Tok}en Padding and Hierarchical Semantic Regularization, designed to fully exploit all tokens for enhanced latent modeling. Specifically, we introduce \textbf{Redundant Token Padding} to activate tail tokens more frequently, thereby alleviating information over-concentration in the early tokens. In addition, we apply \textbf{Hierarchical Semantic Regularization} to align the decoding features of earlier tokens with those from a pre-trained vision foundation model, while progressively reducing the regularization strength toward the tail to allow finer low-level detail reconstruction. Extensive experiments demonstrate the effectiveness of ReTok: on ImageNet 256$\times$256, our method achieves superior generation performance compared with both flexible and fixed-length tokenizers. Code will be available at: \href{https://github.com/zfu006/ReTok}{https://github.com/zfu006/ReTok}