What value do explicit high level concepts have in vision to language problems?

TL;DR

This paper investigates the impact of explicitly incorporating high-level semantic concepts into CNN-RNN models for vision-to-language tasks, demonstrating significant performance improvements in image captioning and VQA.

Contribution

It introduces a method to embed high-level semantic concepts into CNN-RNN models and shows that external semantic information further enhances performance.

Findings

Explicit high-level concepts improve V2L model accuracy.

External semantic information further boosts performance.

The approach advances state-of-the-art results in captioning and VQA.

Abstract

Much of the recent progress in Vision-to-Language (V2L) problems has been achieved through a combination of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). This approach does not explicitly represent high-level semantic concepts, but rather seeks to progress directly from image features to text. We propose here a method of incorporating high-level concepts into the very successful CNN-RNN approach, and show that it achieves a significant improvement on the state-of-the-art performance in both image captioning and visual question answering. We also show that the same mechanism can be used to introduce external semantic information and that doing so further improves performance. In doing so we provide an analysis of the value of high level semantic information in V2L problems.