Mirai: Autoregressive Visual Generation Needs Foresight

TL;DR

Mirai introduces a foresight-based training framework for autoregressive visual generators, significantly enhancing convergence speed and image quality without architectural changes.

Contribution

Mirai demonstrates how injecting future information aligned with internal representations improves autoregressive image generation.

Findings

Mirai accelerates convergence of LlamaGen-B by up to 10×.

Mirai reduces ImageNet generation FID from 5.34 to 4.34.

Foresight aligned with internal representations enhances causal modeling.

Abstract

Autoregressive (AR) visual generators model images as sequences of discrete tokens and are trained with a next-token likelihood objective. This strict causal supervision optimizes each step based only on the immediate next token, which can weaken global coherence and slow convergence. We investigate whether foresight, training signals that originate from later tokens, can improve autoregressive visual generation. We conduct a series of controlled diagnostics along the injection level, foresight layout, and foresight source axes, revealing a key insight: aligning foresight with AR models' internal representations on the 2D image grid improves causal modeling. We formulate this insight with Mirai (meaning "future" in Japanese), a general framework that injects future information into AR training with no architecture change and no extra inference overhead: Mirai-E uses explicit foresight from multiple future positions of unidirectional representations, whereas Mirai-I leverages implicit foresight from matched bidirectional representations. Extensive experiments show that Mirai significantly accelerates convergence and improves generation quality. For instance, Mirai can speed up LlamaGen-B's convergence by up to 10$\times$ and reduce the generation FID from 5.34 to 4.34 on the ImageNet class-condition image generation benchmark. Our study highlights that visual autoregressive models need foresight.