Don't Just Listen, Use Your Imagination: Leveraging Visual Common Sense for Non-Visual Tasks

TL;DR

This paper introduces a method for leveraging visual common sense knowledge from images to improve performance on textual tasks like fill-in-the-blank and paraphrasing, by imagining scenes behind the text.

Contribution

It proposes a novel approach to incorporate visual common sense into textual reasoning tasks through scene imagination, outperforming text-only baselines.

Findings

Outperforms strong text-only baselines on the proposed tasks

Introduces new benchmarks for evaluating common sense reasoning beyond recognition

Provides publicly available code and datasets for further research

Abstract

Artificial agents today can answer factual questions. But they fall short on questions that require common sense reasoning. Perhaps this is because most existing common sense databases rely on text to learn and represent knowledge. But much of common sense knowledge is unwritten - partly because it tends not to be interesting enough to talk about, and partly because some common sense is unnatural to articulate in text. While unwritten, it is not unseen. In this paper we leverage semantic common sense knowledge learned from images - i.e. visual common sense - in two textual tasks: fill-in-the-blank and visual paraphrasing. We propose to "imagine" the scene behind the text, and leverage visual cues from the "imagined" scenes in addition to textual cues while answering these questions. We imagine the scenes as a visual abstraction. Our approach outperforms a strong text-only baseline on these tasks. Our proposed tasks can serve as benchmarks to quantitatively evaluate progress in solving tasks that go "beyond recognition". Our code and datasets are publicly available.