Visual Reasoning with Natural Language

TL;DR

This paper introduces a natural language visual reasoning task where agents must determine the truth of descriptive statements about images, aiming to bridge the gap between language complexity and visual understanding.

Contribution

It proposes a new task for natural language visual reasoning and presents a synthetic dataset, with ongoing work on collecting real-world visual data.

Findings

Synthetic image corpus developed

Task formulated for language-vision reasoning

Work on real-world data collection in progress

Abstract

Natural language provides a widely accessible and expressive interface for robotic agents. To understand language in complex environments, agents must reason about the full range of language inputs and their correspondence to the world. Such reasoning over language and vision is an open problem that is receiving increasing attention. While existing data sets focus on visual diversity, they do not display the full range of natural language expressions, such as counting, set reasoning, and comparisons. We propose a simple task for natural language visual reasoning, where images are paired with descriptive statements. The task is to predict if a statement is true for the given scene. This abstract describes our existing synthetic images corpus and our current work on collecting real vision data.