I Know About "Up"! Enhancing Spatial Reasoning in Visual Language Models Through 3D Reconstruction

TL;DR

This paper introduces ZeroVLM, a model that enhances visual language models' spatial reasoning by using 3D reconstruction and prompting, significantly improving accuracy on spatial reasoning tasks.

Contribution

The paper presents ZeroVLM, a novel approach combining 3D reconstruction and prompting to improve spatial reasoning in visual language models.

Findings

Up to 19.48% accuracy improvement on spatial reasoning datasets

Effective use of 3D reconstruction for visual reasoning

Prompting mechanism enhances spatial understanding

Abstract

Visual Language Models (VLMs) are essential for various tasks, particularly visual reasoning tasks, due to their robust multi-modal information integration, visual reasoning capabilities, and contextual awareness. However, existing \VLMs{}' visual spatial reasoning capabilities are often inadequate, struggling even with basic tasks such as distinguishing left from right. To address this, we propose the \ours{} model, designed to enhance the visual spatial reasoning abilities of VLMS. ZeroVLM employs Zero-1-to-3, a 3D reconstruction model for obtaining different views of the input images and incorporates a prompting mechanism to further improve visual spatial reasoning. Experimental results on four visual spatial reasoning datasets show that our \ours{} achieves up to 19.48% accuracy improvement, which indicates the effectiveness of the 3D reconstruction and prompting mechanisms of our ZeroVLM.