MGD-GAN: Text-to-Pedestrian generation through Multi-Grained Discrimination

TL;DR

This paper introduces MGD-GAN, a novel generative model that synthesizes detailed pedestrian images from text descriptions by using multi-grained discrimination and attention mechanisms, improving realism and diversity.

Contribution

The paper proposes a multi-grained discrimination framework with human-part-based and global discriminators, along with a word-level attention mechanism for fine details in text-to-pedestrian synthesis.

Findings

Significant improvement in generation quality and diversity metrics.

Effective capture of complex pedestrian body structures.

Superior performance demonstrated on CUHK Person Description Dataset.

Abstract

In this paper, we investigate the problem of text-to-pedestrian synthesis, which has many potential applications in art, design, and video surveillance. Existing methods for text-to-bird/flower synthesis are still far from solving this fine-grained image generation problem, due to the complex structure and heterogeneous appearance that the pedestrians naturally take on. To this end, we propose the Multi-Grained Discrimination enhanced Generative Adversarial Network, that capitalizes a human-part-based Discriminator (HPD) and a self-cross-attended (SCA) global Discriminator in order to capture the coherence of the complex body structure. A fined-grained word-level attention mechanism is employed in the HPD module to enforce diversified appearance and vivid details. In addition, two pedestrian generation metrics, named Pose Score and Pose Variance, are devised to evaluate the generation quality and diversity, respectively. We conduct extensive experiments and ablation studies on the caption-annotated pedestrian dataset, CUHK Person Description Dataset. The substantial improvement over the various metrics demonstrates the efficacy of MGD-GAN on the text-to-pedestrian synthesis scenario.