Spatial Traces: Enhancing VLA Models with Spatial-Temporal Understanding

TL;DR

This paper introduces a novel visual prompting method that integrates spatial and temporal understanding in vision-language-action models, significantly improving task success rates with minimal training data.

Contribution

The paper presents a new approach that combines spatial and temporal cues via visual traces, enhancing VLA models' performance in virtual environments.

Findings

4% increase in task success over SpatialVLA

19% increase over TraceVLA

Effective with minimal training data

Abstract

Vision-Language-Action models have demonstrated remarkable capabilities in predicting agent movements within virtual environments and real-world scenarios based on visual observations and textual instructions. Although recent research has focused on enhancing spatial and temporal understanding independently, this paper presents a novel approach that integrates both aspects through visual prompting. We introduce a method that projects visual traces of key points from observations onto depth maps, enabling models to capture both spatial and temporal information simultaneously. The experiments in SimplerEnv show that the mean number of tasks successfully solved increased for 4% compared to SpatialVLA and 19% compared to TraceVLA. Furthermore, we show that this enhancement can be achieved with minimal training data, making it particularly valuable for real-world applications where data collection is challenging. The project page is available at https://ampiromax.github.io/ST-VLA.