4D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration

TL;DR

The paper introduces 4D-VLA, a pretraining approach that integrates 4D spatiotemporal information and memory bank sampling to improve robotic vision-language-action tasks, demonstrating superior performance in simulation and real-world tests.

Contribution

It proposes a novel 4D information integration method with memory bank sampling to enhance pretraining efficiency and spatial reasoning in robotic vision-language models.

Findings

Significant success rate improvement over OpenVLA.

Enhanced spatial perception and generalization demonstrated.

Outperforms existing methods in simulated and real-world experiments.

Abstract

Leveraging diverse robotic data for pretraining remains a critical challenge. Existing methods typically model the dataset's action distribution using simple observations as inputs. However, these inputs are often incomplete, resulting in a dispersed conditional action distribution-an issue we refer to as coordinate system chaos and state chaos. This inconsistency significantly hampers pretraining efficiency. To address this, we propose 4D-VLA, a novel approach that effectively integrates 4D information into the input to mitigate these sources of chaos. Our model introduces depth and temporal information into visual features with sequential RGB-D inputs, aligning the coordinate systems of the robot and the scene. This alignment endows the model with strong spatiotemporal reasoning capabilities while minimizing training overhead. Additionally, we introduce memory bank sampling, a frame sampling strategy designed to extract informative frames from historical images, further improving effectiveness and efficiency. Experimental results demonstrate that our pretraining method and architectural components substantially enhance model performance. In both simulated and real-world experiments, our model achieves a significant increase in success rate over OpenVLA. To further assess spatial perception and generalization to novel views, we introduce MV-Bench, a multi-view simulation benchmark. Our model consistently outperforms existing methods, demonstrating stronger spatial understanding and adaptability.