SFTok: Bridging the Performance Gap in Discrete Tokenizers

TL;DR

SFTok is a novel discrete image tokenizer that employs a multi-step iterative process with self-forcing and debiasing strategies, achieving state-of-the-art reconstruction quality and efficient class-to-image generation at high compression rates.

Contribution

We introduce SFTok, a discrete tokenizer with a multi-step iterative mechanism and self-forcing guidance, addressing training-inference inconsistency and improving image reconstruction quality.

Findings

Achieves state-of-the-art image reconstruction (rFID=1.21) on ImageNet.

Performs exceptionally well in class-to-image generation (gFID=2.29).

Operates effectively at a high compression rate of 64 tokens per image.

Abstract

Recent advances in multimodal models highlight the pivotal role of image tokenization in high-resolution image generation. By compressing images into compact latent representations, tokenizers enable generative models to operate in lower-dimensional spaces, thereby improving computational efficiency and reducing complexity. Discrete tokenizers naturally align with the autoregressive paradigm but still lag behind continuous ones, limiting their adoption in multimodal systems. To address this, we propose \textbf{SFTok}, a discrete tokenizer that incorporates a multi-step iterative mechanism for precise reconstruction. By integrating \textbf{self-forcing guided visual reconstruction} and \textbf{debias-and-fitting training strategy}, SFTok resolves the training-inference inconsistency in multi-step process, significantly enhancing image reconstruction quality. At a high compression rate of only 64 tokens per image, SFTok achieves state-of-the-art reconstruction quality on ImageNet (rFID = 1.21) and demonstrates exceptional performance in class-to-image generation tasks (gFID = 2.29).