LLaVA$^3$: Representing 3D Scenes like a Cubist Painter to Boost 3D Scene Understanding of VLMs

TL;DR

LLaVA$^3$ enhances 3D scene understanding in vision-language models by using multi-view 2D images and omnidirectional representations inspired by Cubist art, without requiring 3D training data or fine-tuning.

Contribution

It introduces a novel approach that leverages multi-view 2D images and intermediate 3D reconstructions to improve 3D scene understanding in VLMs without additional training.

Findings

Outperforms previous 2D-based VLM solutions in 3D VQA.

Effective in 3D language grounding tasks.

Operates without fine-tuning or 3D training data.

Abstract

Developing a multi-modal language model capable of understanding 3D scenes remains challenging due to the limited availability of 3D training data, in contrast to the abundance of 2D datasets used for vision-language models (VLM). As an alternative, we introduce LLaVA$^3$ (pronounced LLaVA-Cube), a novel method that improves the 3D scene understanding capabilities of VLM using only multi-view 2D images and without any fine-tuning. Inspired by Cubist painters, who represented multiple viewpoints of a 3D object within a single picture, we propose to describe the 3D scene for the VLM through omnidirectional visual representations of each object. These representations are derived from an intermediate multi-view 3D reconstruction of the scene. Extensive experiments on 3D VQA and 3D language grounding show that our approach outperforms previous 2D-based VLM solutions.