Defining and Evaluating Visual Language Models' Basic Spatial Abilities: A Perspective from Psychometrics

TL;DR

This paper introduces a psychometric framework for evaluating basic spatial abilities in Visual Language Models, benchmarking 13 models and revealing gaps compared to human spatial cognition, with implications for developing more embodied AI.

Contribution

It pioneers a psychometric approach to define and measure five basic spatial abilities in VLMs, providing a diagnostic toolkit and methodological foundation for spatial intelligence development.

Findings

VLMs show hierarchical spatial abilities similar to humans

Smaller models outperform larger ones in spatial tasks

Interventions like chain-of-thought improve VLM performance modestly

Abstract

The Theory of Multiple Intelligences underscores the hierarchical nature of cognitive capabilities. To advance Spatial Artificial Intelligence, we pioneer a psychometric framework defining five Basic Spatial Abilities (BSAs) in Visual Language Models (VLMs): Spatial Perception, Spatial Relation, Spatial Orientation, Mental Rotation, and Spatial Visualization. Benchmarking 13 mainstream VLMs through nine validated psychometric experiments reveals significant gaps versus humans (average score 24.95 vs. 68.38), with three key findings: 1) VLMs mirror human hierarchies (strongest in 2D orientation, weakest in 3D rotation) with independent BSAs (Pearson's r<0.4); 2) Smaller models such as Qwen2-VL-7B surpass larger counterparts, with Qwen leading (30.82) and InternVL2 lagging (19.6); 3) Interventions like chain-of-thought (0.100 accuracy gain) and 5-shot training (0.259 improvement) show limits from architectural constraints. Identified barriers include weak geometry encoding and missing dynamic simulation. By linking psychometric BSAs to VLM capabilities, we provide a diagnostic toolkit for spatial intelligence evaluation, methodological foundations for embodied AI development, and a cognitive science-informed roadmap for achieving human-like spatial intelligence.