Hita: Holistic Tokenizer for Autoregressive Image Generation

TL;DR

Hita is a new holistic image tokenizer designed for autoregressive models, improving global understanding, training speed, and image quality in tasks like generation, style transfer, and in-painting.

Contribution

Hita introduces a holistic-to-local tokenization scheme with learnable holistic queries and a sequential structure, enhancing autoregressive image generation.

Findings

Achieves 2.59 FID and 281.9 IS on ImageNet

Accelerates training speed of AR generators

Effective in zero-shot style transfer and in-painting

Abstract

Vanilla autoregressive image generation models generate visual tokens step-by-step, limiting their ability to capture holistic relationships among token sequences. Moreover, because most visual tokenizers map local image patches into latent tokens, global information is limited. To address this, we introduce \textit{Hita}, a novel image tokenizer for autoregressive (AR) image generation. It introduces a holistic-to-local tokenization scheme with learnable holistic queries and local patch tokens. Hita incorporates two key strategies to better align with the AR generation process: 1) {arranging} a sequential structure with holistic tokens at the beginning, followed by patch-level tokens, and using causal attention to maintain awareness of previous tokens; and 2) adopting a lightweight fusion module before feeding the de-quantized tokens into the decoder to control information flow and prioritize holistic tokens. Extensive experiments show that Hita accelerates the training speed of AR generators and outperforms those trained with vanilla tokenizers, achieving \textbf{2.59 FID} and \textbf{281.9 IS} on the ImageNet benchmark. Detailed analysis of the holistic representation highlights its ability to capture global image properties, such as textures, materials, and shapes. Additionally, Hita also demonstrates effectiveness in zero-shot style transfer and image in-painting. The code is available at \href{https://github.com/CVMI-Lab/Hita}{https://github.com/CVMI-Lab/Hita}.