Language Models for Image Captioning: The Quirks and What Works

TL;DR

This paper compares two recent image captioning methods using the same CNN, analyzes their issues, and combines their strengths to achieve new state-of-the-art results on COCO, though human judgment improvements are limited.

Contribution

It provides a direct comparison of CNN-based language models for captioning and introduces a hybrid approach that improves benchmark scores.

Findings

Hybrid approach achieves new record on COCO dataset

BLEU scores improve but human judgments do not

Analysis of linguistic irregularities and caption repetition

Abstract

Two recent approaches have achieved state-of-the-art results in image captioning. The first uses a pipelined process where a set of candidate words is generated by a convolutional neural network (CNN) trained on images, and then a maximum entropy (ME) language model is used to arrange these words into a coherent sentence. The second uses the penultimate activation layer of the CNN as input to a recurrent neural network (RNN) that then generates the caption sequence. In this paper, we compare the merits of these different language modeling approaches for the first time by using the same state-of-the-art CNN as input. We examine issues in the different approaches, including linguistic irregularities, caption repetition, and data set overlap. By combining key aspects of the ME and RNN methods, we achieve a new record performance over previously published results on the benchmark COCO dataset. However, the gains we see in BLEU do not translate to human judgments.