Benchmarking Spatiotemporal Reasoning in LLMs and Reasoning Models: Capabilities and Challenges

TL;DR

This paper introduces STARK, a benchmark for evaluating the spatiotemporal reasoning capabilities of LLMs and LRMs across diverse tasks, revealing strengths and limitations in geometric and world-knowledge reasoning.

Contribution

The paper presents a comprehensive hierarchical benchmark, STARK, for systematically assessing and comparing LLMs and LRMs in complex spatiotemporal reasoning tasks.

Findings

LLMs show limited success in geometric reasoning tasks as complexity increases

LRMs demonstrate robust performance, often surpassing traditional methods

Performance gap narrows in world-knowledge reasoning, with some LLMs outperforming LRMs

Abstract

Spatiotemporal reasoning plays a key role in Cyber-Physical Systems (CPS). Despite advances in Large Language Models (LLMs) and Large Reasoning Models (LRMs), their capacity to reason about complex spatiotemporal signals remains underexplored. This paper proposes a hierarchical SpatioTemporal reAsoning benchmaRK, STARK, to systematically evaluate LLMs across three levels of reasoning complexity: state estimation (e.g., predicting field variables, localizing and tracking events in space and time), spatiotemporal reasoning over states (e.g., inferring spatial-temporal relationships), and world-knowledge-aware reasoning that integrates contextual and domain knowledge (e.g., intent prediction, landmark-aware navigation). We curate 26 distinct spatiotemporal tasks with diverse sensor modalities, comprising 14,552 challenges where models answer directly or by Python Code Interpreter. Evaluating 3 LRMs and 8 LLMs, we find LLMs achieve limited success in tasks requiring geometric reasoning (e.g., multilateration or triangulation), particularly as complexity increases. Surprisingly, LRMs show robust performance across tasks with various levels of difficulty, often competing or surpassing traditional first-principle-based methods. Our results show that in reasoning tasks requiring world knowledge, the performance gap between LLMs and LRMs narrows, with some LLMs even surpassing LRMs. However, the LRM o3 model continues to achieve leading performance across all evaluated tasks, a result attributed primarily to the larger size of the reasoning models. STARK motivates future innovations in model architectures and reasoning paradigms for intelligent CPS by providing a structured framework to identify limitations in the spatiotemporal reasoning of LLMs and LRMs.