Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoning

TL;DR

Selftok introduces a novel discrete visual tokenizer that employs autoregressive modeling and diffusion processes, enabling effective reinforcement learning and high-quality image representation without relying on spatial priors.

Contribution

It proposes Selftok, a discrete visual tokenizer with autoregressive properties using diffusion, unifying vision-language modeling and reinforcement learning capabilities.

Findings

Selftok achieves state-of-the-art reconstruction quality and compression.

Reinforcement learning with Selftok significantly improves visual generation performance.

Selftok enables training vision-language models without text-image pairs.

Abstract

We completely discard the conventional spatial prior in image representation and introduce a novel discrete visual tokenizer: Self-consistency Tokenizer (Selftok). At its design core, we compose an autoregressive (AR) prior -- mirroring the causal structure of language -- into visual tokens by using the reverse diffusion process of image generation. The AR property makes Selftok fundamentally distinct from traditional spatial tokens in the following two key ways: - Selftok offers an elegant and minimalist approach to unify diffusion and AR for vision-language models (VLMs): By representing images with Selftok tokens, we can train a VLM using a purely discrete autoregressive architecture -- like that in LLMs -- without requiring additional modules or training objectives. - We theoretically show that the AR prior satisfies the Bellman equation, whereas the spatial prior does not. Therefore, Selftok supports reinforcement learning (RL) for visual generation with effectiveness comparable to that achieved in LLMs. Besides the AR property, Selftok is also a SoTA tokenizer that achieves a favorable trade-off between high-quality reconstruction and compression rate. We use Selftok to build a pure AR VLM for both visual comprehension and generation tasks. Impressively, without using any text-image training pairs, a simple policy gradient RL working in the visual tokens can significantly boost the visual generation benchmark, surpassing all the existing models by a large margin. Therefore, we believe that Selftok effectively addresses the long-standing challenge that visual tokens cannot support effective RL. When combined with the well-established strengths of RL in LLMs, this brings us one step closer to realizing a truly multimodal LLM. Project Page: https://selftok-team.github.io/report/.