iFSQ: Improving FSQ for Image Generation with 1 Line of Code

TL;DR

This paper introduces iFSQ, a simple yet effective modification to Finite Scalar Quantization that improves image generation modeling by ensuring optimal bin use and reconstruction accuracy, enabling better benchmarking and insights into AR and diffusion models.

Contribution

The authors propose iFSQ, a one-line code change to FSQ that guarantees optimal quantization and reconstruction, and use it to analyze and compare autoregressive and diffusion models.

Findings

Optimal bits per dimension for discrete-continuous balance is around 4 bits.

AR models converge faster initially, but diffusion models outperform in the long run.

iFSQ provides a unified benchmark for different image generation models.

Abstract

The field of image generation is currently bifurcated into autoregressive (AR) models operating on discrete tokens and diffusion models utilizing continuous latents. This divide, rooted in the distinction between VQ-VAEs and VAEs, hinders unified modeling and fair benchmarking. Finite Scalar Quantization (FSQ) offers a theoretical bridge, yet vanilla FSQ suffers from a critical flaw: its equal-interval quantization can cause activation collapse. This mismatch forces a trade-off between reconstruction fidelity and information efficiency. In this work, we resolve this dilemma by simply replacing the activation function in original FSQ with a distribution-matching mapping to enforce a uniform prior. Termed iFSQ, this simple strategy requires just one line of code yet mathematically guarantees both optimal bin utilization and reconstruction precision. Leveraging iFSQ as a controlled benchmark, we uncover two key insights: (1) The optimal equilibrium between discrete and continuous representations lies at approximately 4 bits per dimension. (2) Under identical reconstruction constraints, AR models exhibit rapid initial convergence, whereas diffusion models achieve a superior performance ceiling, suggesting that strict sequential ordering may limit the upper bounds of generation quality. Finally, we extend our analysis by adapting Representation Alignment (REPA) to AR models, yielding LlamaGen-REPA. Codes is available at https://github.com/Tencent-Hunyuan/iFSQ