Learning Discrete Autoregressive Priors with Wasserstein Gradient Flow

TL;DR

This paper introduces wAR-Tok, a new tokenizer training method that aligns token distributions with autoregressive priors using Wasserstein gradient flow, improving generation quality on CIFAR-10 and ImageNet.

Contribution

It proposes a distribution-level prior-matching approach during tokenizer training, addressing prior mismatch issues in two-stage discrete image modeling.

Findings

wAR-Tok reduces autoregressive loss.

Improves generation FID on CIFAR-10 and ImageNet.

Maintains reconstruction quality while enhancing prior alignment.

Abstract

Discrete image tokenizers are commonly trained in two stages: first for reconstruction, and then with a prior model fitted to the frozen token sequences. This decoupling leaves the tokenizer unaware of the model that will later generate its tokens. As a result, the learned tokens may preserve image information well but still be difficult for an autoregressive (AR) prior to predict from left to right. We analyze this mismatch using Tripartite Variational Consistency (TVC), which decomposes latent-variable learning into three consistency conditions: conditional-likelihood consistency, prior consistency, and posterior consistency. TVC shows that two-stage training preserves the reconstruction side but leaves prior consistency outside the tokenizer objective: the overall token distribution is fixed before the AR prior participates in training. Motivated by this view, we add a distribution-level prior-matching signal during tokenizer training, while keeping the reconstruction objective unchanged. We optimize this signal with a Wasserstein-gradient-flow update. For hard categorical tokens, the update reduces to a token-level contrast between an auxiliary AR model that tracks the tokenizer's current token distribution and the target AR prior. It requires only forward passes through the two AR models and does not backpropagate through either of them. The resulting tokenizer, wAR-Tok, reduces AR loss and improves generation FID on CIFAR-10 and ImageNet at comparable reconstruction quality.