Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Study

TL;DR

This paper introduces Spatial Intelligence Grid (SIG), a structured scene representation that improves the evaluation and learning of visual-spatial intelligence in foundation models, especially for autonomous driving applications.

Contribution

The paper proposes SIG as a novel grid-based schema for explicit spatial encoding, along with SIG-informed metrics and a new benchmark, SIGBench, to enhance spatial reasoning in foundation models.

Findings

SIG improves VSI metric performance across models.

SIG yields more stable and comprehensive spatial reasoning.

SIGBench provides extensive annotated data for autonomous driving scenarios.

Abstract

How to integrate and verify spatial intelligence in foundation models remains an open challenge. Current practice often proxies Visual-Spatial Intelligence (VSI) with purely textual prompts and VQA-style scoring, which obscures geometry, invites linguistic shortcuts, and weakens attribution to genuinely spatial skills. We introduce Spatial Intelligence Grid (SIG): a structured, grid-based schema that explicitly encodes object layouts, inter-object relations, and physically grounded priors. As a complementary channel to text, SIG provides a faithful, compositional representation of scene structure for foundation-model reasoning. Building on SIG, we derive SIG-informed evaluation metrics that quantify a model's intrinsic VSI, which separates spatial capability from language priors. In few-shot in-context learning with state-of-the-art multimodal LLMs (e.g. GPT- and Gemini-family models), SIG yields consistently larger, more stable, and more comprehensive gains across all VSI metrics compared to VQA-only representations, indicating its promise as a data-labeling and training schema for learning VSI. We also release SIGBench, a benchmark of 1.4K driving frames annotated with ground-truth SIG labels and human gaze traces, supporting both grid-based machine VSI tasks and attention-driven, human-like VSI tasks in autonomous-driving scenarios.