Spatio-Temporal LLM: Reasoning about Environments and Actions

TL;DR

This paper introduces the REA dataset and two spatio-temporal LLM models to improve understanding of environments and actions in multimodal prompts, addressing a key challenge in real-world agent reasoning.

Contribution

The paper presents a large-scale REA dataset and two novel spatio-temporal LLM baselines, advancing the ability of models to reason about environments and actions from multimodal data.

Findings

STLLM baselines outperform existing models on REA

Models effectively fuse point cloud, video, and text data

Spatio-temporal understanding improves agent reasoning capabilities

Abstract

Despite significant recent progress of Multimodal Large Language Models (MLLMs), current MLLMs are challenged by "spatio-temporal" prompts, i.e., prompts that refer to 1) the entirety of an environment encoded in a point cloud that the MLLM should consider; and simultaneously also refer to 2) actions that happened in part of the environment and are encoded in a short ego-centric video clip. However, such a holistic spatio-temporal understanding is important for agents operating in the real world. To address this challenge, we first develop a framework to collect a large-scale dataset. Using the collected "Reasoning about Environments and Actions" (REA) dataset, we show that recent MLLMs indeed struggle to correctly answer "spatio-temporal" prompts. Building on this dataset, we study two spatio-temporal LLM (STLLM) baselines: 1) STLLM-3D, which directly fuses point cloud, video, and text representations as inputs to the LLM; and 2) STLLM-Aligner, which aligns spatial context with video and text before LLM decoding. Both baselines aim to enhance spatial understanding of environments and temporal grounding of egocentric observations. On REA, the STLLM baselines outperform existing models, demonstrating the effectiveness of our designs. Code and data are available at https://zoezheng126.github.io/STLLM-website/.