Towards Accurate Image Coding: Improved Autoregressive Image Generation with Dynamic Vector Quantization

TL;DR

This paper introduces a dynamic vector quantization framework for autoregressive image generation, improving accuracy and efficiency by encoding image regions with variable-length codes and generating images from coarse to fine details.

Contribution

It proposes a novel DQ-VAE for variable-length encoding and a DQ-Transformer for coarse-to-fine autoregressive image generation, addressing limitations of fixed-length coding.

Findings

Outperforms existing models in quality and speed

Effective in various image generation tasks

Reduces redundancy and improves detail representation

Abstract

Existing vector quantization (VQ) based autoregressive models follow a two-stage generation paradigm that first learns a codebook to encode images as discrete codes, and then completes generation based on the learned codebook. However, they encode fixed-size image regions into fixed-length codes and ignore their naturally different information densities, which results in insufficiency in important regions and redundancy in unimportant ones, and finally degrades the generation quality and speed. Moreover, the fixed-length coding leads to an unnatural raster-scan autoregressive generation. To address the problem, we propose a novel two-stage framework: (1) Dynamic-Quantization VAE (DQ-VAE) which encodes image regions into variable-length codes based on their information densities for an accurate and compact code representation. (2) DQ-Transformer which thereby generates images autoregressively from coarse-grained (smooth regions with fewer codes) to fine-grained (details regions with more codes) by modeling the position and content of codes in each granularity alternately, through a novel stacked-transformer architecture and shared-content, non-shared position input layers designs. Comprehensive experiments on various generation tasks validate our superiorities in both effectiveness and efficiency. Code will be released at https://github.com/CrossmodalGroup/DynamicVectorQuantization.