Can adversarial training learn image captioning ?

TL;DR

This paper explores the feasibility of using adversarial training, specifically a conditional GAN architecture without pre-training or reinforcement, for generating image captions from images.

Contribution

It introduces a novel adversarial architecture for image captioning that does not rely on pre-training or reinforcement learning, and discusses its evaluation and interpretability.

Findings

First adversarial approach to image captioning without pre-training

Provides evaluation framework for adversarial captioning models

Demonstrates potential of GANs in natural language generation for images

Abstract

Recently, generative adversarial networks (GAN) have gathered a lot of interest. Their efficiency in generating unseen samples of high quality, especially images, has improved over the years. In the field of Natural Language Generation (NLG), the use of the adversarial setting to generate meaningful sentences has shown to be difficult for two reasons: the lack of existing architectures to produce realistic sentences and the lack of evaluation tools. In this paper, we propose an adversarial architecture related to the conditional GAN (cGAN) that generates sentences according to a given image (also called image captioning). This attempt is the first that uses no pre-training or reinforcement methods. We also explain why our experiment settings can be safely evaluated and interpreted for further works.