Depth Adaptive Efficient Visual Autoregressive Modeling

TL;DR

DepthVAR introduces an adaptive, computation-efficient framework for visual autoregressive modeling by dynamically allocating processing depth per token, significantly accelerating inference with minimal quality loss.

Contribution

It presents a novel, training-free method that adaptively assigns computational depth to tokens, surpassing traditional pruning techniques in efficiency and quality.

Findings

Achieves 2.3× to 3.1× acceleration with minimal quality loss.

Outperforms existing hard-pruning methods in compute-performance trade-offs.

Demonstrates effectiveness across high-resolution image generation tasks.

Abstract

Visual Autoregressive (VAR) modeling inefficiently applies a fixed computational depth to each position when generating high-resolution images. While existing methods accelerate inference by pruning tokens using frequency maps, their binary hard-pruning approach is fundamentally limited and fails to improve quality even with better frequency estimation. Observing that VAR models possess significant depth redundancy, we propose a paradigm shift from pruning entire tokens to adaptively allocating per-token computational depth. To this end, we introduce DepthVAR, a training-free framework that dynamically allocates computation. It integrates an adaptive depth scheduler, which assigns computational depth via a cyclic rotated schedule for balanced, non-static refinement, with a dynamic inference process that translates these depths into layer-major masks, selectively applies transformer blocks, and blends the resulting codes to ensure each token's influence is proportional to its processing depth. Extensive experiments show that DepthVAR achieves 2.3$\times$-3.1$\times$ acceleration with minimal quality loss, offering a competitive compute-performance trade-off compared to existing hard-pruning approaches. Code is available at https://github.com/STOVAGtz/DepthVAR