STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence

TL;DR

STAR-Bench introduces a comprehensive audio benchmark that evaluates deep spatio-temporal reasoning in sound, revealing significant gaps in current models' perceptual and reasoning abilities compared to humans.

Contribution

The paper formalizes audio 4D intelligence and creates STAR-Bench, a novel benchmark combining perceptual and holistic reasoning tasks with high-quality data curation methods.

Findings

Models show large performance drops on STAR-Bench tasks.

Open-source models lag behind humans in perception and reasoning.

Current models are bottlenecked by fine-grained perceptual understanding.

Abstract

Despite rapid progress in Multi-modal Large Language Models and Large Audio-Language Models, existing audio benchmarks largely test semantics that can be recovered from text captions, masking deficits in fine-grained perceptual reasoning. We formalize audio 4D intelligence that is defined as reasoning over sound dynamics in time and 3D space, and introduce STAR-Bench to measure it. STAR-Bench combines a Foundational Acoustic Perception setting (six attributes under absolute and relative regimes) with a Holistic Spatio-Temporal Reasoning setting that includes segment reordering for continuous and discrete processes and spatial tasks spanning static localization, multi-source relations, and dynamic trajectories. Our data curation pipeline uses two methods to ensure high-quality samples. For foundational tasks, we use procedurally synthesized and physics-simulated audio. For holistic data, we follow a four-stage process that includes human annotation and final selection based on human performance. Unlike prior benchmarks where caption-only answering reduces accuracy slightly, STAR-Bench induces far larger drops (-31.5\% temporal, -35.2\% spatial), evidencing its focus on linguistically hard-to-describe cues. Evaluating 19 models reveals substantial gaps compared with humans and a capability hierarchy: closed-source models are bottlenecked by fine-grained perception, while open-source models lag across perception, knowledge, and reasoning. Our STAR-Bench provides critical insights and a clear path forward for developing future models with a more robust understanding of the physical world.