STT-Arena: A More Realistic Environment for Tool-Using with Spatio-Temporal Dynamics

TL;DR

STT-Arena is a new benchmark with 227 realistic interactive tasks designed to evaluate and improve large language models' ability to adapt to spatio-temporal disruptions in dynamic environments.

Contribution

The paper introduces STT-Arena, a comprehensive benchmark for spatio-temporal dynamic reasoning, and proposes an iterative refinement and RL approach to enhance LLM performance.

Findings

State-of-the-art models achieve less than 40% accuracy on STT-Arena.

Identified three common failure modes: Stale-State Execution, Misdiagnosis, Missing Verification.

Proposed STT-Agent-4B outperforms existing models on the benchmark.

Abstract

Large language models (LLMs) deployed in real-world agentic applications must be capable of replanning and adapting when mid-task disruptions invalidate their prior decisions. Existing dynamic benchmarks primarily measure whether LLMs can detect temporal changes in a timely manner, leaving the complementary challenge of adaptive replanning under spatio-temporal dynamics largely unexplored. We introduce STT-Arena (Spatio-Temporal Tool-Use Arena), a benchmark of 227 high-quality interactive tasks spanning nine spatio-temporal conflict types and four solvability levels. Each task is grounded in a realistic, executable environment equipped with injected spatio-temporal triggers that can abruptly invalidate an ongoing plan, forcing the model to detect the state shift and construct a revised execution strategy. Extensive evaluation of frontier LLMs reveals that even the SOTA proprietary models, including Claude-4.6-Opus, achieves less than 40\% overall accuracies, highlighting the fundamental difficulty of spatio-temporal dynamic reasoning. Systematic analysis of failure trajectories uncovers three recurring error modes of existing models: Stale-State Execution, Misdiagnosis of Dynamic Triggers, and Missing Post-Adaptation Verification. Guided by these findings, we propose an iterative trajectory refinement technique that eliminates these failure patterns from training data, and combine it with online RL to produce STT-Agent-4B which outperforms frontier LLMs on STT-Arena.