HiST-VLA: A Hierarchical Spatio-Temporal Vision-Language-Action Model for End-to-End Autonomous Driving

TL;DR

HiST-VLA is a hierarchical spatio-temporal vision-language-action model that significantly improves autonomous driving by enhancing spatial reasoning, efficiency, and command grounding, leading to state-of-the-art results on key benchmarks.

Contribution

The paper introduces a novel hierarchical spatio-temporal VLA framework with dynamic token sparsification and a transformer-based planner for end-to-end autonomous driving.

Findings

Achieves 88.6 EPDMS on Navtest benchmark.

Attains 50.9 EPDMS on Navhard benchmark.

Demonstrates state-of-the-art performance on NAVSIM v2.

Abstract

Vision-Language-Action (VLA) models offer promising capabilities for autonomous driving through multimodal understanding. However, their utilization in safety-critical scenarios is constrained by inherent limitations, including imprecise numerical reasoning, weak 3D spatial awareness, and high sensitivity to context. To address these challenges, we propose HiST-VLA, a novel Hierarchical Spatio-Temporal VLA model designed for reliable trajectory generation. Our framework enhances 3D spatial and temporal reasoning by integrating geometric awareness with fine-grained driving commands and state history prompting. To ensure computational efficiency, we integrate dynamic token sparsification into the VLA architecture. This approach fuses redundant tokens rather than filtering them, effectively reducing redundancy without sacrificing model performance. Furthermore, we employ a hierarchical transformer-based planner to progressively refine coarse VLA waypoints into fine-grained trajectories. Crucially, the planner utilizes dynamic latent regularization to incorporate language commands, ensuring strict spatial grounding and temporal coherence. Extensive evaluation on the NAVSIM v2 benchmark demonstrates state-of-the-art performance on Navtest, achieving an EPDMS of 88.6, and EPDMS of 50.9 on pseudo closed-loop Navhard benchmark.