Instella-T2I: Pushing the Limits of 1D Discrete Latent Space Image Generation

TL;DR

This paper introduces 1D binary image latents for high-resolution image generation, significantly reducing token count while maintaining quality, enabling faster training and inference with competitive results.

Contribution

The paper presents a novel 1D binary latent space for image tokenization, achieving high-resolution image generation with fewer tokens and improved efficiency compared to traditional methods.

Findings

Achieves competitive performance with only 128 tokens for 1024x1024 images.

Reduces token count by up to 32 times compared to VQ-VAEs.

Enables large batch training on single GPU nodes within 200 GPU days.

Abstract

Image tokenization plays a critical role in reducing the computational demands of modeling high-resolution images, significantly improving the efficiency of image and multimodal understanding and generation. Recent advances in 1D latent spaces have reduced the number of tokens required by eliminating the need for a 2D grid structure. In this paper, we further advance compact discrete image representation by introducing 1D binary image latents. By representing each image as a sequence of binary vectors, rather than using traditional one-hot codebook tokens, our approach preserves high-resolution details while maintaining the compactness of 1D latents. To the best of our knowledge, our text-to-image models are the first to achieve competitive performance in both diffusion and auto-regressive generation using just 128 discrete tokens for images up to 1024x1024, demonstrating up to a 32-fold reduction in token numbers compared to standard VQ-VAEs. The proposed 1D binary latent space, coupled with simple model architectures, achieves marked improvements in speed training and inference speed. Our text-to-image models allow for a global batch size of 4096 on a single GPU node with 8 AMD MI300X GPUs, and the training can be completed within 200 GPU days. Our models achieve competitive performance compared to modern image generation models without any in-house private training data or post-training refinements, offering a scalable and efficient alternative to conventional tokenization methods.