Co-VisiON: Co-Visibility ReasONing on Sparse Image Sets of Indoor Scenes

TL;DR

This paper introduces the Co-VisiON benchmark to evaluate co-visibility reasoning in sparse indoor scenes, revealing current models' limitations and proposing a new multi-view baseline inspired by human cognition.

Contribution

The paper presents a new benchmark dataset and evaluation for co-visibility reasoning, and introduces a novel multi-view baseline model that improves over existing vision-only approaches.

Findings

Current models struggle with sparse co-visibility reasoning.

A proprietary vision-language model outperforms vision-only baselines.

The proposed Covis model narrows the gap to human performance.

Abstract

Humans exhibit a remarkable ability to recognize co-visibility-the 3D regions simultaneously visible in multiple images-even when these images are sparsely distributed across a complex scene. This ability is foundational to 3D vision, robotic perception, and relies not only on low-level feature matching but also on high-level spatial reasoning and cognitive integration. Yet, it remains unclear whether current vision models can replicate this human-level proficiency. In this work, we introduce the Co-VisiON benchmark, designed to evaluate human-inspired co-visibility reasoning across more than 1,000 sparse-view indoor scenarios. Our results show that while co-visibility is often approached as a low-level feature-matching task, it remains challenging for existing vision models under sparse conditions. Notably, a proprietary vision-language model surpasses all vision-only baselines, but all models fall significantly short of human performance. This gap underscores the limitations of current architectures and motivates the need for models that integrate spatial and semantic information in a human-like manner. Inspired by human visual cognition, we propose a novel multi-view baseline, Covis, which achieves top performance among pure vision models and narrows the gap to the proprietary VLM. We hope our benchmark and findings will spur further advancements in developing vision models capable of robust, cognitively inspired reasoning in challenging, sparse environments. Our dataset and source code can be found at https://ai4ce.github.io/CoVISION.