Learning to generalize to new compositions in image understanding

TL;DR

This paper compares structured representations and recurrent neural networks for image captioning, demonstrating that structured models generalize better to new scene compositions and proposing a benchmark for compositional generalization.

Contribution

It introduces structured representations for image captioning, showing improved generalization to new compositions over LSTM-based models, and advocates for compositional benchmarks.

Findings

Structured models outperform LSTMs in compositional generalization (~7x accuracy)

Structured representations enable quantification of generalization to unseen combinations

Proposes compositional splits as a benchmark for image captioning

Abstract

Recurrent neural networks have recently been used for learning to describe images using natural language. However, it has been observed that these models generalize poorly to scenes that were not observed during training, possibly depending too strongly on the statistics of the text in the training data. Here we propose to describe images using short structured representations, aiming to capture the crux of a description. These structured representations allow us to tease-out and evaluate separately two types of generalization: standard generalization to new images with similar scenes, and generalization to new combinations of known entities. We compare two learning approaches on the MS-COCO dataset: a state-of-the-art recurrent network based on an LSTM (Show, Attend and Tell), and a simple structured prediction model on top of a deep network. We find that the structured model generalizes to new compositions substantially better than the LSTM, ~7 times the accuracy of predicting structured representations. By providing a concrete method to quantify generalization for unseen combinations, we argue that structured representations and compositional splits are a useful benchmark for image captioning, and advocate compositional models that capture linguistic and visual structure.