ManiGAN: Text-Guided Image Manipulation

TL;DR

ManiGAN is a novel generative adversarial network designed for semantically editing images based on text descriptions, effectively manipulating specific attributes while preserving irrelevant content, validated on CUB and COCO datasets.

Contribution

The paper introduces ManiGAN, a new GAN with a text-image affine combination module and detail correction module for improved text-guided image manipulation.

Findings

Outperforms existing methods on CUB and COCO datasets

Effectively manipulates specified attributes while preserving irrelevant content

Proposes a new evaluation metric for image manipulation quality

Abstract

The goal of our paper is to semantically edit parts of an image matching a given text that describes desired attributes (e.g., texture, colour, and background), while preserving other contents that are irrelevant to the text. To achieve this, we propose a novel generative adversarial network (ManiGAN), which contains two key components: text-image affine combination module (ACM) and detail correction module (DCM). The ACM selects image regions relevant to the given text and then correlates the regions with corresponding semantic words for effective manipulation. Meanwhile, it encodes original image features to help reconstruct text-irrelevant contents. The DCM rectifies mismatched attributes and completes missing contents of the synthetic image. Finally, we suggest a new metric for evaluating image manipulation results, in terms of both the generation of new attributes and the reconstruction of text-irrelevant contents. Extensive experiments on the CUB and COCO datasets demonstrate the superior performance of the proposed method. Code is available at https://github.com/mrlibw/ManiGAN.