Communicating about Space: Language-Mediated Spatial Integration Across Partial Views

TL;DR

This paper introduces COSMIC, a benchmark for evaluating multimodal language models' ability to develop shared spatial understanding through dialogue in 3D environments, revealing current limitations and room for improvement.

Contribution

The paper presents COSMIC, a new benchmark for collaborative spatial communication, and systematically evaluates MLLMs' capabilities in forming shared spatial mental models.

Findings

MLLMs reliably identify shared anchor objects across views.

MLLMs perform poorly on relational reasoning and global map building.

Human dialogues achieve 95% accuracy, while best models reach 72%, indicating significant gap.

Abstract

Humans build shared spatial understanding by communicating partial, viewpoint-dependent observations. We ask whether Multimodal Large Language Models (MLLMs) can do the same, aligning distinct egocentric views through dialogue to form a coherent, allocentric mental model of a shared environment. To study this systematically, we introduce COSMIC, a benchmark for Collaborative Spatial Communication. In this setting, two static MLLM agents observe a 3D indoor environment from different viewpoints and exchange natural-language messages to solve spatial queries. COSMIC contains 899 diverse scenes and 1250 question-answer pairs spanning five tasks. We find a capability hierarchy, MLLMs are most reliable at identifying shared anchor objects across views, perform worse on relational reasoning, and largely fail at building globally consistent maps, performing near chance, even for frontier models. Moreover, we find thinking capability yields gains in anchor grounding, but is insufficient for higher-level spatial communication. To contextualize model behavior, we collect 250 human-human dialogues. Humans achieve 95% aggregate accuracy, while the best model, Gemini-3-Pro-Thinking, reaches 72%, leaving substantial room for improvement. Moreover, human conversations grow more precise as partners align on a shared spatial understanding, whereas MLLMs keep exploring without converging, suggesting limited capacity to form and sustain a robust shared mental model throughout the dialogue. Our code and data is available at https://github.com/ankursikarwar/Cosmic.