DAE-GAN: Dynamic Aspect-aware GAN for Text-to-Image Synthesis

TL;DR

This paper introduces DAE-GAN, a novel text-to-image synthesis model that effectively utilizes aspect information at multiple granularities to improve image realism and semantic accuracy.

Contribution

The paper proposes a dynamic aspect-aware GAN with a new multi-granularity text representation and an aspect-aware refinement module for enhanced image synthesis.

Findings

Outperforms existing methods on CUB-200 and COCO datasets.

Effectively incorporates aspect information for detailed image generation.

Achieves higher semantic consistency and visual quality.

Abstract

Text-to-image synthesis refers to generating an image from a given text description, the key goal of which lies in photo realism and semantic consistency. Previous methods usually generate an initial image with sentence embedding and then refine it with fine-grained word embedding. Despite the significant progress, the 'aspect' information (e.g., red eyes) contained in the text, referring to several words rather than a word that depicts 'a particular part or feature of something', is often ignored, which is highly helpful for synthesizing image details. How to make better utilization of aspect information in text-to-image synthesis still remains an unresolved challenge. To address this problem, in this paper, we propose a Dynamic Aspect-awarE GAN (DAE-GAN) that represents text information comprehensively from multiple granularities, including sentence-level, word-level, and aspect-level. Moreover, inspired by human learning behaviors, we develop a novel Aspect-aware Dynamic Re-drawer (ADR) for image refinement, in which an Attended Global Refinement (AGR) module and an Aspect-aware Local Refinement (ALR) module are alternately employed. AGR utilizes word-level embedding to globally enhance the previously generated image, while ALR dynamically employs aspect-level embedding to refine image details from a local perspective. Finally, a corresponding matching loss function is designed to ensure the text-image semantic consistency at different levels. Extensive experiments on two well-studied and publicly available datasets (i.e., CUB-200 and COCO) demonstrate the superiority and rationality of our method.