Visual Implicit Autoregressive Modeling

TL;DR

VIAR introduces an implicit equilibrium layer in visual autoregressive models, enabling efficient, high-quality image generation with adjustable compute and memory, outperforming many existing methods.

Contribution

The paper proposes VIAR, a novel implicit autoregressive model with Jacobian-Free Backpropagation, reducing memory and increasing efficiency while maintaining or surpassing state-of-the-art quality.

Findings

VIAR achieves FID 2.16 on ImageNet 256x256.

VIAR reduces peak memory from 19.24 GB to 8.53 GB.

VIAR doubles throughput to 32.08 images/sec on a single RTX 4090.

Abstract

Visual Autoregressive Modeling (VAR) based on next-scale prediction achieves strong generation quality, but their explicit deep stacks fix the amount of computation per scale and inflate memory at high resolutions. We introduce Visual Implicit Autoregressive Modeling (VIAR), a next-scale autoregressive generator that embeds an implicit equilibrium layer between shallow pre/post blocks. The implicit layer is trained with Jacobian-Free Backpropagation, yielding constant training memory, while inference exposes a per-scale iteration knob that enables compute control. On ImageNet 256x256 benchmark, VIAR attains FID 2.16, and sFID 8.07 with only 38.4% parameters of VAR, matching or surpassing strong AR baselines and remaining competitive with large diffusion models. By controlling the per-scale knob, VIAR can reduce peak memory from 19.24 GB to 8.53 GB and doubles throughput from 15.16 to 32.08 images/s on a single RTX 4090, without retraining. Ablations show that fewer steps are sufficient for fixed-point iterations to converge and that VIAR consistently dominates VAR across quality efficiency operating points. In zero shot in-painting and class-conditional editing, VIAR produces sharper details and smoother boundaries while preserving global structure, validating the benefits of implicit equilibria and per-scale compute control for practical, deployable visual generation.