On the Generalization Capacities of MLLMs for Spatial Intelligence

TL;DR

This paper introduces a camera-aware framework for multimodal large language models to improve their spatial reasoning and generalization across different camera setups by disentangling camera parameters from scene understanding.

Contribution

The paper proposes a novel camera-aware MLLM framework that incorporates camera intrinsics, data augmentation, and geometric priors to enhance spatial reasoning and generalization.

Findings

Camera-aware MLLMs outperform naive models in cross-camera tests.

Disentangling camera parameters improves spatial reasoning.

Camera-awareness is essential for robust spatial intelligence.

Abstract

Multimodal Large Language Models (MLLMs) that directly process RGB inputs for tasks like 3D localization and navigation have shown remarkable potential. However, we argue that these RGB-only approaches are fundamentally flawed in their ability to generalize across cameras. By ignoring camera parameters, they entangle an object's physical properties with the camera's perspective, creating an irresolvable ambiguity. We show this leads MLLMs to overfit to the training camera distribution, rather than learning true and generalizable 3D geometric principles. To address this, we propose Camera-Aware MLLM framework for spatial MLLMs. It learns generalizable spatial reasoning by: (i) injecting camera intrinsics via a dense embedding that conditions each visual token; (ii) introducing a camera-aware data augmentation strategy that synthetically varies camera parameters, forcing the model to disentangle camera properties from scene content; and (iii) distilling geometric priors from a 3D vision foundation model. Extensive experiments demonstrate that camera-aware MLLMs substantially outperform their naive counterparts, particularly in cross-camera generalization tests on spatially-grounded tasks, indicating that camera-awareness is not only beneficial but also a prerequisite for robust and generalizable spatial intelligence in MLLMs.