Spatio-Temporal LLM: Reasoning about Environments and Actions
Haozhen Zheng, Beitong Tian, Mingyuan Wu, Zhenggang Tang, Klara Nahrstedt, Alex Schwing

TL;DR
This paper introduces the REA dataset and two spatio-temporal LLM models to improve understanding of environments and actions in multimodal prompts, addressing a key challenge in real-world agent reasoning.
Contribution
The paper presents a large-scale REA dataset and two novel spatio-temporal LLM baselines, advancing the ability of models to reason about environments and actions from multimodal data.
Findings
STLLM baselines outperform existing models on REA
Models effectively fuse point cloud, video, and text data
Spatio-temporal understanding improves agent reasoning capabilities
Abstract
Despite significant recent progress of Multimodal Large Language Models (MLLMs), current MLLMs are challenged by "spatio-temporal" prompts, i.e., prompts that refer to 1) the entirety of an environment encoded in a point cloud that the MLLM should consider; and simultaneously also refer to 2) actions that happened in part of the environment and are encoded in a short ego-centric video clip. However, such a holistic spatio-temporal understanding is important for agents operating in the real world. To address this challenge, we first develop a framework to collect a large-scale dataset. Using the collected "Reasoning about Environments and Actions" (REA) dataset, we show that recent MLLMs indeed struggle to correctly answer "spatio-temporal" prompts. Building on this dataset, we study two spatio-temporal LLM (STLLM) baselines: 1) STLLM-3D, which directly fuses point cloud, video, and text…
Peer Reviews
Decision·Submitted to ICLR 2026
* Addresses an important and underexplored problem in MLLMs: joint spatial and temporal reasoning. * Introduces a new dataset (REA) that is built with careful attention to grounding and scene structure, using real-world environments and realistic tasks.
* All evaluations are performed in kitchen scenarios. The models may not generalize to other environments such as offices or outdoor scenes. * The analysis does not deeply examine what the models actually learn. Are they truly reasoning over space and time, or simply picking up on surface correlations? * The data pipeline relies on noisy pose and segmentation estimates, but the paper lacks analysis of annotation errors or their impact on dataset quality.
1. The paper empirically demonstrates that existing MLLMs perform poorly at spatio-temporal tasks (23.85% to 31.46% across tasks), whereas the REA-trained baselines achieve 41.89% overall accuracy. 1. The generation of rich, fine-grained question-answer pairs from the EPIC-Kitchens dataset is well thought through; in particular, the point-cloud reconstruction (Section 3.1.2) describes the generation of 3D views from a variety of scenes, which are then used to produce the REA questions. Prior w
1. The REA dataset is not compared against existing benchmarks for embodied agents, such as Embodied-Bench [1], which covers both high-level and low-level motor tasks in 4 environments (unlike the REA benchmark, which focuses on the EPIC kitchen environment). It is thus unclear what the novelty of the new benchmark is: is it the 3D-based point cloud reconstruction (which is missing in Embodied-Bench)? More concretely: a good benchmark should ideally demonstrate that models trained on it can tra
1. The primary strength is the introduction of the REA dataset. It identifies a clear weakness in existing MLLMs and provides a new, challenging benchmark with a detailed data collection pipeline (Sec 3.1) to spur further research. 2. The paper proposes two simple but effective baseline architectures (STLLM-3D and STLLM-Aligner) that demonstrate the clear benefit of fusing point cloud data with video-language models.
1. The paper's definition of "spatio-temporal" reasoning—understanding an "entirety of an environment encoded in a point cloud" and "actions... encoded in a short ego-centric video" —feels like an overclaim. This formulation primarily models a single agent's movement and interaction within an otherwise static 3D space. The essence of spatio-temporal reasoning, which would distinguish it from spatial reasoning, should arguably involve understanding the dynamics of other moving objects, agents, or
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Topic Modeling · Constraint Satisfaction and Optimization
