Draft-and-Revise: Effective Image Generation with Contextual RQ-Transformer

TL;DR

This paper introduces a novel image generation framework called Draft-and-Revise with Contextual RQ-Transformer, which considers global image context during generation, leading to state-of-the-art results in conditional image synthesis.

Contribution

The paper proposes a new Draft-and-Revise decoding method with Contextual RQ-Transformer that effectively incorporates global context and improves image quality and diversity.

Findings

Achieves state-of-the-art results on conditional image generation.

Effectively controls quality-diversity trade-off in image synthesis.

Demonstrates improved global context reflection in generated images.

Abstract

Although autoregressive models have achieved promising results on image generation, their unidirectional generation process prevents the resultant images from fully reflecting global contexts. To address the issue, we propose an effective image generation framework of Draft-and-Revise with Contextual RQ-transformer to consider global contexts during the generation process. As a generalized VQ-VAE, RQ-VAE first represents a high-resolution image as a sequence of discrete code stacks. After code stacks in the sequence are randomly masked, Contextual RQ-Transformer is trained to infill the masked code stacks based on the unmasked contexts of the image. Then, Contextual RQ-Transformer uses our two-phase decoding, Draft-and-Revise, and generates an image, while exploiting the global contexts of the image during the generation process. Specifically. in the draft phase, our model first focuses on generating diverse images despite rather low quality. Then, in the revise phase, the model iteratively improves the quality of images, while preserving the global contexts of generated images. In experiments, our method achieves state-of-the-art results on conditional image generation. We also validate that the Draft-and-Revise decoding can achieve high performance by effectively controlling the quality-diversity trade-off in image generation.