Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models

TL;DR

Loc3R-VLM introduces a novel framework that enhances vision-language models with 3D spatial reasoning from monocular videos, enabling improved localization and question-answering by grounding perception and language in 3D space.

Contribution

It presents a new approach combining global layout reconstruction and explicit situation modeling to provide direct 3D spatial supervision for vision-language models.

Findings

Achieves state-of-the-art results in language-based localization.

Outperforms existing methods on 3D question-answering benchmarks.

Demonstrates effective 3D understanding through spatial supervision.

Abstract

Multimodal Large Language Models (MLLMs) have made impressive progress in connecting vision and language, but they still struggle with spatial understanding and viewpoint-aware reasoning. Recent efforts aim to augment the input representations with geometric cues rather than explicitly teaching models to reason in 3D space. We introduce Loc3R-VLM, a framework that equips 2D Vision-Language Models with advanced 3D understanding capabilities from monocular video input. Inspired by human spatial cognition, Loc3R-VLM relies on two joint objectives: global layout reconstruction to build a holistic representation of the scene structure, and explicit situation modeling to anchor egocentric perspective. These objectives provide direct spatial supervision that grounds both perception and language in a 3D context. To ensure geometric consistency and metric-scale alignment, we leverage lightweight camera pose priors extracted from a pre-trained 3D foundation model. Loc3R-VLM achieves state-of-the-art performance in language-based localization and outperforms existing 2D- and video-based approaches on situated and general 3D question-answering benchmarks, demonstrating that our spatial supervision framework enables strong 3D understanding. Project page: https://kevinqu7.github.io/loc3r-vlm