LiteVAR: Compressing Visual Autoregressive Modelling with Efficient Attention and Quantization

TL;DR

LiteVAR introduces a set of efficiency techniques including attention reduction and quantization to significantly decrease computational and memory requirements of visual autoregressive models, enabling deployment on resource-limited devices with minimal performance loss.

Contribution

The paper proposes novel efficient attention and quantization methods for VAR models, reducing resource consumption while preserving image generation quality.

Findings

85.2% reduction in attention computation

50% reduction in overall memory usage

1.5x latency reduction

Abstract

Visual Autoregressive (VAR) has emerged as a promising approach in image generation, offering competitive potential and performance comparable to diffusion-based models. However, current AR-based visual generation models require substantial computational resources, limiting their applicability on resource-constrained devices. To address this issue, we conducted analysis and identified significant redundancy in three dimensions of the VAR model: (1) the attention map, (2) the attention outputs when using classifier free guidance, and (3) the data precision. Correspondingly, we proposed efficient attention mechanism and low-bit quantization method to enhance the efficiency of VAR models while maintaining performance. With negligible performance lost (less than 0.056 FID increase), we could achieve 85.2% reduction in attention computation, 50% reduction in overall memory and 1.5x latency reduction. To ensure deployment feasibility, we developed efficient training-free compression techniques and analyze the deployment feasibility and efficiency gain of each technique.