SpaRRTa: A Synthetic Benchmark for Evaluating Spatial Intelligence in Visual Foundation Models

TL;DR

SpaRRTa introduces a benchmark for assessing spatial reasoning in visual foundation models, revealing disparities in their ability to understand object positions, and aims to guide future improvements in spatial awareness capabilities.

Contribution

The paper presents SpaRRTa, a novel synthetic benchmark for evaluating spatial relation recognition in VFMs, addressing limitations of existing 3D tasks and providing diverse, annotated images for comprehensive assessment.

Findings

Significant disparities in spatial reasoning abilities among state-of-the-art VFMs.

SpaRRTa effectively generates diverse, photorealistic images with controllable object arrangements.

Insights into mechanisms that support or hinder spatial awareness in VFMs.

Abstract

Visual Foundation Models (VFMs), such as DINO and CLIP, excel in semantic understanding of images but exhibit limited spatial reasoning capabilities, which limits their applicability to embodied systems. As a result, recent work incorporates some 3D tasks (such as depth estimation) into VFM training. However, VFM performance remains inconsistent across other spatial tasks, raising the question of whether these models truly have spatial awareness or overfit to specific 3D objectives. To address this question, we introduce the Spatial Relation Recognition Task (SpaRRTa) benchmark, which evaluates the ability of VFMs to identify relative positions of objects in the image. Unlike traditional 3D objectives that focus on precise metric prediction (e.g., surface normal estimation), SpaRRTa probes a fundamental capability underpinning more advanced forms of human-like spatial understanding. SpaRRTa generates an arbitrary number of photorealistic images with diverse scenes and fully controllable object arrangements, along with freely accessible spatial annotations. Evaluating a range of state-of-the-art VFMs, we reveal significant disparities between their spatial reasoning abilities. Through our analysis, we provide insights into the mechanisms that support or hinder spatial awareness in modern VFMs. We hope that SpaRRTa will serve as a useful tool for guiding the development of future spatially aware visual models.