Can LLMs See Without Pixels? Benchmarking Spatial Intelligence from Textual Descriptions

TL;DR

This paper introduces SiT-Bench, a benchmark to evaluate Large Language Models' spatial intelligence using textual descriptions, revealing their strengths and limitations in global spatial reasoning without visual input.

Contribution

The paper presents SiT-Bench, a comprehensive textual benchmark for assessing LLMs' spatial reasoning, highlighting the importance of explicit reasoning and providing a new resource for future research.

Findings

LLMs excel in localized semantic tasks

A significant gap exists in global spatial consistency

Explicit spatial reasoning improves LLM performance

Abstract

Recent advancements in Spatial Intelligence (SI) have predominantly relied on Vision-Language Models (VLMs), yet a critical question remains: does spatial understanding originate from visual encoders or the fundamental reasoning backbone? Inspired by this question, we introduce SiT-Bench, a novel benchmark designed to evaluate the SI performance of Large Language Models (LLMs) without pixel-level input, comprises over 3,800 expert-annotated items across five primary categories and 17 subtasks, ranging from egocentric navigation and perspective transformation to fine-grained robotic manipulation. By converting single/multi-view scenes into high-fidelity, coordinate-aware textual descriptions, we challenge LLMs to perform symbolic textual reasoning rather than visual pattern matching. Evaluation results of state-of-the-art (SOTA) LLMs reveals that while models achieve proficiency in localized semantic tasks, a significant "spatial gap" remains in global consistency. Notably, we find that explicit spatial reasoning significantly boosts performance, suggesting that LLMs possess latent world-modeling potential. Our proposed dataset SiT-Bench serves as a foundational resource to foster the development of spatially-grounded LLM backbones for future VLMs and embodied agents. Our code and benchmark will be released at https://github.com/binisalegend/SiT-Bench .