Multimodal Language Models Cannot Spot Spatial Inconsistencies

TL;DR

This paper demonstrates that current multimodal large language models struggle to detect spatial inconsistencies across different views of scenes, highlighting their limited understanding of 3D geometry.

Contribution

The authors introduce a new task and a scalable method to evaluate MLLMs' ability to identify spatial inconsistencies, revealing their significant shortcomings.

Findings

MLLMs underperform compared to humans in detecting spatial inconsistencies

Model performance varies greatly across different scene attributes

Current models have a fragile understanding of 3D structure

Abstract

Spatial consistency is a fundamental property of the visual world and a key requirement for models that aim to understand physical reality. Despite recent advances, multimodal large language models (MLLMs) often struggle to reason about 3D geometry across multiple views. Rather than asking models to describe scene attributes, we introduce a more challenging task: given two views of the same scene, identify the object that violates 3D motion consistency. We propose a simple and scalable method for generating realistic, spatially inconsistent image pairs from multi-view scenes, enabling systematic evaluation of this capability. Our results show that state-of-the-art MLLMs significantly underperform human observers and exhibit substantial variability across different scene attributes, revealing a fragile and incomplete understanding of 3D structure. We hope our findings underscore the need for approaches that develop a more deeply grounded understanding of the physical world.