SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer

TL;DR

SoftVQ-VAE introduces a continuous image tokenizer that enhances representation capacity and significantly accelerates image generation, achieving high-quality results with fewer tokens and improved efficiency in generative models.

Contribution

It presents SoftVQ-VAE, a novel continuous tokenizer leveraging soft categorical posteriors, enabling efficient, high-capacity image representation and faster generation in Transformer-based models.

Findings

Achieves up to 18x faster inference for 256x256 images.

Reduces training iterations by 2.3x while maintaining quality.

Maintains competitive FID scores with fewer tokens.

Abstract

Efficient image tokenization with high compression ratios remains a critical challenge for training generative models. We present SoftVQ-VAE, a continuous image tokenizer that leverages soft categorical posteriors to aggregate multiple codewords into each latent token, substantially increasing the representation capacity of the latent space. When applied to Transformer-based architectures, our approach compresses 256x256 and 512x512 images using as few as 32 or 64 1-dimensional tokens. Not only does SoftVQ-VAE show consistent and high-quality reconstruction, more importantly, it also achieves state-of-the-art and significantly faster image generation results across different denoising-based generative models. Remarkably, SoftVQ-VAE improves inference throughput by up to 18x for generating 256x256 images and 55x for 512x512 images while achieving competitive FID scores of 1.78 and 2.21 for SiT-XL. It also improves the training efficiency of the generative models by reducing the number of training iterations by 2.3x while maintaining comparable performance. With its fully-differentiable design and semantic-rich latent space, our experiment demonstrates that SoftVQ-VAE achieves efficient tokenization without compromising generation quality, paving the way for more efficient generative models. Code and model are released.