3DAxiesPrompts: Unleashing the 3D Spatial Task Capabilities of GPT-4V

TL;DR

This paper introduces 3DAxiesPrompts, a novel visual prompting method that enhances GPT-4V's ability to perform 3D spatial tasks by creating a 3D coordinate system and annotated scale information, enabling precise 3D spatial reasoning.

Contribution

The paper presents a new visual prompting technique, 3DAxiesPrompts, specifically designed to unlock GPT-4V's capabilities in 3D spatial understanding and tasks.

Findings

GPT-4V with 3DAP can accurately perform 3D spatial tasks

The method improves 2D to 3D point reconstruction and matching

Experimental results validate the effectiveness of 3DAP on the proposed dataset

Abstract

In this work, we present a new visual prompting method called 3DAxiesPrompts (3DAP) to unleash the capabilities of GPT-4V in performing 3D spatial tasks. Our investigation reveals that while GPT-4V exhibits proficiency in discerning the position and interrelations of 2D entities through current visual prompting techniques, its abilities in handling 3D spatial tasks have yet to be explored. In our approach, we create a 3D coordinate system tailored to 3D imagery, complete with annotated scale information. By presenting images infused with the 3DAP visual prompt as inputs, we empower GPT-4V to ascertain the spatial positioning information of the given 3D target image with a high degree of precision. Through experiments, We identified three tasks that could be stably completed using the 3DAP method, namely, 2D to 3D Point Reconstruction, 2D to 3D point matching, and 3D Object Detection. We perform experiments on our proposed dataset 3DAP-Data, the results from these experiments validate the efficacy of 3DAP-enhanced GPT-4V inputs, marking a significant stride in 3D spatial task execution.