Enhancing the Spatial Awareness Capability of Multi-Modal Large Language Model

TL;DR

This paper improves the spatial awareness of multi-modal large language models by integrating precise geometric and scene graph information, leading to better performance in spatial understanding tasks across various benchmarks.

Contribution

The paper introduces a method that incorporates detailed spatial position data and scene graphs to enhance MLLM's spatial reasoning capabilities.

Findings

Significant improvement in spatial awareness tasks

Enhanced performance on benchmarks like MME and MM-Vet

Effective integration of geometric and scene graph data

Abstract

The Multi-Modal Large Language Model (MLLM) refers to an extension of the Large Language Model (LLM) equipped with the capability to receive and infer multi-modal data. Spatial awareness stands as one of the crucial abilities of MLLM, encompassing diverse skills related to understanding spatial relationships among objects and between objects and the scene area. Industries such as autonomous driving, smart healthcare, robotics, virtual, and augmented reality heavily demand MLLM's spatial awareness capabilities. However, there exists a noticeable gap between the current spatial awareness capabilities of MLLM and the requirements set by human needs. To address this issue, this paper proposes using more precise spatial position information between objects to guide MLLM in providing more accurate responses to user-related inquiries. Specifically, for a particular multi-modal task, we utilize algorithms for acquiring geometric spatial information and scene graphs to obtain relevant geometric spatial information and scene details of objects involved in the query. Subsequently, based on this information, we direct MLLM to address spatial awareness-related queries posed by the user. Extensive experiments were conducted in benchmarks such as MME, MM-Vet, and other multi-modal large language models. The experimental results thoroughly confirm the efficacy of the proposed method in enhancing the spatial awareness tasks and associated tasks of MLLM.