GeoGuess: Multimodal Reasoning based on Hierarchy of Visual Information in Street View

TL;DR

This paper introduces GeoGuess, a challenging multimodal reasoning task involving identifying street view locations and explaining visual clues, supported by a new dataset and a hierarchical reasoning method called SightSense.

Contribution

It proposes a novel task, GeoGuess, emphasizing hierarchical visual reasoning, and provides a new dataset GeoExplain along with the SightSense method for improved multimodal reasoning.

Findings

SightSense achieves outstanding performance on GeoGuess.

The GeoExplain dataset effectively supports hierarchical visual reasoning.

GeoGuess advances understanding of multimodal reasoning with hierarchical visual clues.

Abstract

Multimodal reasoning is a process of understanding, integrating and inferring information across different data modalities. It has recently attracted surging academic attention as a benchmark for Artificial Intelligence (AI). Although there are various tasks for evaluating multimodal reasoning ability, they still have limitations. Lack of reasoning on hierarchical visual clues at different levels of granularity, e.g., local details and global context, is of little discussion, despite its frequent involvement in real scenarios. To bridge the gap, we introduce a novel and challenging task for multimodal reasoning, namely GeoGuess. Given a street view image, the task is to identify its location and provide a detailed explanation. A system that succeeds in GeoGuess should be able to detect tiny visual clues, perceive the broader landscape, and associate with vast geographic knowledge. Therefore, GeoGuess would require the ability to reason between hierarchical visual information and geographic knowledge. In this work, we establish a benchmark for GeoGuess by introducing a specially curated dataset GeoExplain which consists of panoramas-geocoordinates-explanation tuples. Additionally, we present a multimodal and multilevel reasoning method, namely SightSense which can make prediction and generate comprehensive explanation based on hierarchy of visual information and external knowledge. Our analysis and experiments demonstrate their outstanding performance in GeoGuess.