CVP: Central-Peripheral Vision-Inspired Multimodal Model for Spatial Reasoning

TL;DR

CVP is a multimodal model inspired by human central and peripheral vision, enhancing spatial reasoning in 3D scene understanding by explicitly modeling scene structure and context.

Contribution

Introduces a novel central-peripheral vision-inspired framework with target-affinity tokens and allocentric grids for improved spatial reasoning.

Findings

Achieves state-of-the-art results on 3D scene understanding benchmarks.

Effectively captures global scene context and spatial arrangements.

Enhances structured, context-aware understanding of complex environments.

Abstract

We present a central-peripheral vision-inspired framework (CVP), a simple yet effective multimodal model for spatial reasoning that draws inspiration from the two types of human visual fields -- central vision and peripheral vision. Existing approaches primarily rely on unstructured representations, such as point clouds, voxels, or patch features, and inject scene context implicitly via coordinate embeddings. However, this often results in limited spatial reasoning capabilities due to the lack of explicit, high-level structural understanding. To address this limitation, we introduce two complementary components into a Large Multimodal Model-based architecture: target-affinity token, analogous to central vision, that guides the model's attention toward query-relevant objects; and allocentric grid, akin to peripheral vision, that captures global scene context and spatial arrangements. These components work in tandem to enable structured, context-aware understanding of complex 3D environments. Experiments show that CVP achieves state-of-the-art performance across a range of 3D scene understanding benchmarks.