See, Remember, Explore: A Benchmark and Baselines for Streaming Spatial Reasoning

TL;DR

This paper introduces S3-Bench, a comprehensive streaming spatial reasoning benchmark with real-world and simulated data, and proposes AMF-VLM, a model supporting active exploration and memory-efficient inference, advancing embodied agent spatial understanding.

Contribution

It presents S3-Bench for streaming spatial QA with active perception, and proposes AMF-VLM supporting memory folding and active exploration for improved reasoning.

Findings

AMF-VLM improves accuracy by 8.8% on simulated data.

AMF-VLM improves accuracy by 13.3% on real-world data.

Models with active exploration outperform passive baselines.

Abstract

Spatial understanding is fundamental for embodied agents, yet most spatial VLMs and benchmarks remain offline-evaluating post-hoc QA over pre-recorded inputs and overlooking two crucial deployment-critical requirements: long-horizon streaming inference and active perception when the current view is insufficient. To address this gap, we introduce S3-Bench, a benchmark suite for streaming spatial question answering with active exploration, where queries are temporally grounded to specific timestamps and must be answered using only observations available up to that moment. S3-Bench adopts a dual-domain design, combining a scalable simulator with controllable trajectories and exploration actions, and real-world streaming videos that capture practical sensing artifacts for rigorous generalization evaluation. Overall, it spans 10K+ scenes and 26K+ trajectories, with dedicated training (S3-Train) and evaluation (S3-Eval) splits. We further propose AMF-VLM, which supports streaming spatial reasoning under bounded computing via (i) memory folding, which compresses long-horizon observations into compact structured memory, and (ii) active exploration, which outputs explicit actions (e.g. move/rotate/scan) to acquire missing evidence before answering. Extensive experiments demonstrate that, compared to models using identical training data, our approach yields improvements of 8.8% and 13.3% on the simulated and real splits of S3-Eval, respectively, while maintaining competitive transferability to standard spatial benchmarks.