Let Geometry GUIDE: Layer-wise Unrolling of Geometric Priors in Multimodal LLMs

TL;DR

GUIDE introduces a multi-level geometric prior injection framework in multimodal LLMs, enhancing spatial awareness and reasoning by progressively integrating local to global geometric features.

Contribution

The paper proposes a novel progressive geometric priors injection method, GUIDE, that captures multi-granularity features and aligns them with early LLM layers for improved spatial reasoning.

Findings

GUIDE outperforms existing methods on spatial reasoning tasks.

Multi-level sampling captures detailed local and global geometric features.

Context-aware gating improves spatial cue utilization and noise suppression.

Abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable progress in 2D visual tasks but still exhibit limited physical spatial awareness when processing real-world visual streams. Recently, feed-forward geometric foundation models, which implicitly extract geometric priors, have provided a new pathway to address this issue. However, existing geometry-aware MLLMs are predominantly constrained by the paradigm of single deep-layer extraction and input-level fusion. This flattened fusion leads to the loss of local geometric details and causes semantic mismatches in the early layers. To break this bottleneck, we propose GUIDE (Geometric Unrolling Inside MLLM Early-layers), a progressive geometric priors injection framework. GUIDE performs multi-level sampling within the geometric encoder, comprehensively capturing multi-granularity features ranging from local edges to global topologies. Subsequently, we rigorously align and fuse these multi-level geometric priors step-by-step with the early layers of the MLLM. Building upon the injection of multi-granularity geometric information, this design guides the model to progressively learn the 2D-to-3D transitional process. Furthermore, we introduce a context-aware gating that enables the model to fetch requisite spatial cues based on current semantics, thereby maximizing the utilization efficiency of spatial priors and effectively suppressing redundant geometric noise. Extensive experiments demonstrate that GUIDE significantly outperforms existing baselines on multiple complex spatial reasoning and perception tasks, establishing a novel paradigm for integrating 3D geometric priors into large models.