Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources

TL;DR

This paper introduces a flexible visual question answering system that combines image content with external knowledge bases, enabling it to answer complex, natural language questions about images.

Contribution

It presents a novel method that merges image semantic content with external textual knowledge to enhance neural network-based visual question answering.

Findings

Achieved state-of-the-art results on Toronto COCO-QA dataset.

Achieved state-of-the-art results on MS COCO-VQA dataset.

Demonstrated ability to answer questions beyond image content using external knowledge.

Abstract

We propose a method for visual question answering which combines an internal representation of the content of an image with information extracted from a general knowledge base to answer a broad range of image-based questions. This allows more complex questions to be answered using the predominant neural network-based approach than has previously been possible. It particularly allows questions to be asked about the contents of an image, even when the image itself does not contain the whole answer. The method constructs a textual representation of the semantic content of an image, and merges it with textual information sourced from a knowledge base, to develop a deeper understanding of the scene viewed. Priming a recurrent neural network with this combined information, and the submitted question, leads to a very flexible visual question answering approach. We are specifically able to answer questions posed in natural language, that refer to information not contained in the image. We demonstrate the effectiveness of our model on two publicly available datasets, Toronto COCO-QA and MS COCO-VQA and show that it produces the best reported results in both cases.