Take A Step Back: Rethinking the Two Stages in Visual Reasoning

TL;DR

This paper proposes a two-stage framework for visual reasoning, emphasizing separate symbolization and shared reasoning to improve cross-domain generalization across diverse visual tasks.

Contribution

It introduces a novel two-stage approach with separated encoders for symbolization and a shared reasoner, enhancing generalization in visual reasoning tasks.

Findings

Outperforms existing methods on multiple benchmarks

Demonstrates strong cross-domain generalization

Effective on both 2D and 3D visual tasks

Abstract

Visual reasoning, as a prominent research area, plays a crucial role in AI by facilitating concept formation and interaction with the world. However, current works are usually carried out separately on small datasets thus lacking generalization ability. Through rigorous evaluation of diverse benchmarks, we demonstrate the shortcomings of existing ad-hoc methods in achieving cross-domain reasoning and their tendency to data bias fitting. In this paper, we revisit visual reasoning with a two-stage perspective: (1) symbolization and (2) logical reasoning given symbols or their representations. We find that the reasoning stage is better at generalization than symbolization. Thus, it is more efficient to implement symbolization via separated encoders for different data domains while using a shared reasoner. Given our findings, we establish design principles for visual reasoning frameworks following the separated symbolization and shared reasoning. The proposed two-stage framework achieves impressive generalization ability on various visual reasoning tasks, including puzzles, physical prediction, and visual question answering (VQA), encompassing both 2D and 3D modalities. We believe our insights will pave the way for generalizable visual reasoning.