Spatial-LLaVA: Enhancing Large Language Models with Spatial Referring Expressions for Visual Understanding

TL;DR

Spatial-LLaVA is a multimodal large language model enhanced with spatial referring expressions, trained on a new dataset, that significantly improves spatial reasoning in visual understanding tasks, especially in zero-shot scenarios.

Contribution

The paper introduces Spatial-LLaVA and the SUN-Spot v2.0 dataset, enabling better spatial referring expression understanding in multimodal models, surpassing previous methods by 3.15% on a benchmark.

Findings

Outperforms previous methods by 3.15% on zero-shot spatial reasoning

Enables precise understanding of spatial referring expressions

Applicable to autonomous navigation and robotics tasks

Abstract

Multimodal large language models (MLLMs) have demonstrated remarkable abilities in comprehending visual input alongside text input. Typically, these models are trained on extensive data sourced from the internet, which are sufficient for general tasks such as scene understanding and question answering. However, they often underperform on specialized tasks where online data is scarce, such as determining spatial relationships between objects or localizing unique target objects within a group of objects sharing similar features. In response to this challenge, we introduce the SUN-Spot v2.0 dataset1, now comprising a total of 90k image-caption pairs and additional annotations on the landmark objects. Each image-caption pair utilizes Set-of-Marks prompting as an additional indicator, mapping each landmark object in the image to the corresponding object mentioned in the caption. Furthermore, we present Spatial-LLaVA, an MLLM trained on conversational data generated by a state-of-the-art language model using the SUNSpot v2.0 dataset. Our approach ensures a robust alignment between the objects in the images and their corresponding object mentions in the captions, enabling our model to learn spatial referring expressions without bias from the semantic information of the objects. Spatial-LLaVA outperforms previous methods by 3.15% on the zero-shot Visual Spatial Reasoning benchmark dataset. Spatial-LLaVA is specifically designed to precisely understand spatial referring expressions, making it highly applicable for tasks in real-world scenarios such as autonomous navigation and interactive robotics, where precise object recognition is critical.