FlexVAR: Flexible Visual Autoregressive Modeling without Residual Prediction

TL;DR

FlexVAR introduces a novel flexible autoregressive image generation paradigm that enables high-quality, resolution-agnostic, and task-flexible image synthesis without residual prediction, outperforming existing models on benchmark datasets.

Contribution

It proposes FlexVAR, a new autoregressive modeling approach that removes the residual prediction paradigm, allowing for flexible, high-resolution, and multi-task image generation from low-resolution training.

Findings

Outperforms VAR on ImageNet 256x256 with 1.0B parameters.

Achieves state-of-the-art FID scores in zero-shot transfer.

Supports various image-to-image tasks and resolutions.

Abstract

This work challenges the residual prediction paradigm in visual autoregressive modeling and presents FlexVAR, a new Flexible Visual AutoRegressive image generation paradigm. FlexVAR facilitates autoregressive learning with ground-truth prediction, enabling each step to independently produce plausible images. This simple, intuitive approach swiftly learns visual distributions and makes the generation process more flexible and adaptable. Trained solely on low-resolution images ($\leq$ 256px), FlexVAR can: (1) Generate images of various resolutions and aspect ratios, even exceeding the resolution of the training images. (2) Support various image-to-image tasks, including image refinement, in/out-painting, and image expansion. (3) Adapt to various autoregressive steps, allowing for faster inference with fewer steps or enhancing image quality with more steps. Our 1.0B model outperforms its VAR counterpart on the ImageNet 256$\times$256 benchmark. Moreover, when zero-shot transfer the image generation process with 13 steps, the performance further improves to 2.08 FID, outperforming state-of-the-art autoregressive models AiM/VAR by 0.25/0.28 FID and popular diffusion models LDM/DiT by 1.52/0.19 FID, respectively. When transferring our 1.0B model to the ImageNet 512$\times$512 benchmark in a zero-shot manner, FlexVAR achieves competitive results compared to the VAR 2.3B model, which is a fully supervised model trained at 512$\times$512 resolution.