From Prompts to Pavement Through Time: Temporal Grounding in Agentic Scene-to-Plan Reasoning

TL;DR

This paper investigates how temporal conditioning in large language and multimodal models affects scene interpretation and planning in autonomous vehicles, highlighting its limitations and establishing an empirical benchmark.

Contribution

It introduces three temporal planner architectures, evaluates them on a new benchmark, and clarifies the impact and limits of prompt-based temporal grounding.

Findings

Temporal conditioning reshapes reasoning style but does not significantly improve correctness metrics.

Qualitative analysis shows hazard reasoning and strategic divergence in the Sentinel.

Establishes the first empirical benchmark for temporal scene-to-plan reasoning.

Abstract

Recent attempts to support high-level scene interpretation and planning in Autonomous Vehicles (AVs) using ensembles of Large Language Models (LLMs) and Large Multimodal Models (LMMs) continue to treat time as a secondary property. This lack of temporal grounding leads to inconsistencies in reasoning about continuous actions, undermining both safety and interpretability. This work explores whether temporal conditioning within inter-agent communication can preserve or enhance coherence without introducing degradation in semantic or logical consistency. To investigate this, we introduce three planner architectures with progressively increasing temporal integration and evaluate them on curated subsets of the BDD-X dataset using semantic, syntactic, and logical metrics. Results show that while temporal conditioning reshapes reasoning style, it yields no statistically significant improvements in standard NLP-based correctness metrics. However, qualitative analysis reveals predictive hazard reasoning, stable corrective behavior, and strategic divergence in the Sentinel. These findings clarify the limits of prompt-based temporal grounding and establish the first empirical benchmark for temporal scene-to-plan reasoning.