Efficient-VQGAN: Towards High-Resolution Image Generation with Efficient Vision Transformers

TL;DR

Efficient-VQGAN introduces a two-stage, attention-efficient framework for high-resolution image generation that combines local and global attention mechanisms, resulting in faster, higher-quality image synthesis.

Contribution

The paper proposes a novel two-stage framework with local attention-based quantization and combined attention mechanisms, improving efficiency and quality in high-resolution image generation.

Findings

Outperforms previous methods in image quality and resolution

Achieves faster generation speeds

Demonstrates superior reconstruction quality

Abstract

Vector-quantized image modeling has shown great potential in synthesizing high-quality images. However, generating high-resolution images remains a challenging task due to the quadratic computational overhead of the self-attention process. In this study, we seek to explore a more efficient two-stage framework for high-resolution image generation with improvements in the following three aspects. (1) Based on the observation that the first quantization stage has solid local property, we employ a local attention-based quantization model instead of the global attention mechanism used in previous methods, leading to better efficiency and reconstruction quality. (2) We emphasize the importance of multi-grained feature interaction during image generation and introduce an efficient attention mechanism that combines global attention (long-range semantic consistency within the whole image) and local attention (fined-grained details). This approach results in faster generation speed, higher generation fidelity, and improved resolution. (3) We propose a new generation pipeline incorporating autoencoding training and autoregressive generation strategy, demonstrating a better paradigm for image synthesis. Extensive experiments demonstrate the superiority of our approach in high-quality and high-resolution image reconstruction and generation.