Text-to-Image Synthesis Based on Machine Generated Captions

TL;DR

This paper presents a method for text-to-image synthesis that leverages uncaptioned image datasets by generating captions with an image captioning module and then training a conditional GAN on these captions and images.

Contribution

It introduces a novel approach that combines image captioning with GAN training to enable text-based image generation from uncaptioned datasets.

Findings

Preliminary results show promising image generation quality.

The approach effectively uses uncaptioned datasets for text-to-image synthesis.

Comparison with unconditional GAN demonstrates the potential of the method.

Abstract

Text to Image Synthesis refers to the process of automatic generation of a photo-realistic image starting from a given text and is revolutionizing many real-world applications. In order to perform such process it is necessary to exploit datasets containing captioned images, meaning that each image is associated with one (or more) captions describing it. Despite the abundance of uncaptioned images datasets, the number of captioned datasets is limited. To address this issue, in this paper we propose an approach capable of generating images starting from a given text using conditional GANs trained on uncaptioned images dataset. In particular, uncaptioned images are fed to an Image Captioning Module to generate the descriptions. Then, the GAN Module is trained on both the input image and the machine-generated caption. To evaluate the results, the performance of our solution is compared with the results obtained by the unconditional GAN. For the experiments, we chose to use the uncaptioned dataset LSUN bedroom. The results obtained in our study are preliminary but still promising.