DTGAN: Dual Attention Generative Adversarial Networks for Text-to-Image Generation

TL;DR

DTGAN introduces a single-generator, attention-based model for text-to-image synthesis that improves image quality and semantic consistency, reducing complexity and training time compared to multi-stage approaches.

Contribution

The paper proposes DTGAN, a novel single-generator/discriminator framework with attention modules and a new visual loss for improved text-to-image generation.

Findings

Outperforms state-of-the-art multi-stage models on benchmark datasets.

Attention modules effectively localize discriminative regions and capture global visual content.

Enhances image resolution with a new visual loss ensuring vivid shapes and colors.

Abstract

Most existing text-to-image generation methods adopt a multi-stage modular architecture which has three significant problems: 1) Training multiple networks increases the run time and affects the convergence and stability of the generative model; 2) These approaches ignore the quality of early-stage generator images; 3) Many discriminators need to be trained. To this end, we propose the Dual Attention Generative Adversarial Network (DTGAN) which can synthesize high-quality and semantically consistent images only employing a single generator/discriminator pair. The proposed model introduces channel-aware and pixel-aware attention modules that can guide the generator to focus on text-relevant channels and pixels based on the global sentence vector and to fine-tune original feature maps using attention weights. Also, Conditional Adaptive Instance-Layer Normalization (CAdaILN) is presented to help our attention modules flexibly control the amount of change in shape and texture by the input natural-language description. Furthermore, a new type of visual loss is utilized to enhance the image resolution by ensuring vivid shape and perceptually uniform color distributions of generated images. Experimental results on benchmark datasets demonstrate the superiority of our proposed method compared to the state-of-the-art models with a multi-stage framework. Visualization of the attention maps shows that the channel-aware attention module is able to localize the discriminative regions, while the pixel-aware attention module has the ability to capture the globally visual contents for the generation of an image.