Adversarial Semantic Alignment for Improved Image Captions

TL;DR

This paper introduces a semantic alignment approach using conditional GANs for image captioning, proposing new training methods, a semantic evaluation score, and a novel Out of Context test set to improve captioning quality and generalization.

Contribution

It presents a context-aware LSTM captioner with a co-attentive discriminator, compares training methods, and introduces new evaluation tools including a semantic score and Out of Context dataset.

Findings

SCST training yields more stable gradients and better results than Gumbel ST.

The semantic score correlates well with human judgment.

SCST-based training improves performance on OOC and MS-COCO benchmarks.

Abstract

In this paper we study image captioning as a conditional GAN training, proposing both a context-aware LSTM captioner and co-attentive discriminator, which enforces semantic alignment between images and captions. We empirically focus on the viability of two training methods: Self-critical Sequence Training (SCST) and Gumbel Straight-Through (ST) and demonstrate that SCST shows more stable gradient behavior and improved results over Gumbel ST, even without accessing discriminator gradients directly. We also address the problem of automatic evaluation for captioning models and introduce a new semantic score, and show its correlation to human judgement. As an evaluation paradigm, we argue that an important criterion for a captioner is the ability to generalize to compositions of objects that do not usually co-occur together. To this end, we introduce a small captioned Out of Context (OOC) test set. The OOC set, combined with our semantic score, are the proposed new diagnosis tools for the captioning community. When evaluated on OOC and MS-COCO benchmarks, we show that SCST-based training has a strong performance in both semantic score and human evaluation, promising to be a valuable new approach for efficient discrete GAN training.