ST-$\pi$: Structured SpatioTemporal VLA for Robotic Manipulation

TL;DR

ST-$\pi$ introduces a structured spatiotemporal vision-language-action model that explicitly encodes and reasons about sub-tasks, spatial, and temporal boundaries for improved robotic manipulation.

Contribution

The paper presents a novel structured framework combining spatiotemporal encoding and dual-generator guidance for explicit reasoning in robotic manipulation tasks.

Findings

Effective spatiotemporal reasoning improves manipulation accuracy.

Explicit planning of sub-tasks enhances multi-step task performance.

The model outperforms existing methods on the proposed dataset.

Abstract

Vision-language-action (VLA) models have achieved great success on general robotic tasks, but still face challenges in fine-grained spatiotemporal manipulation. Typically, existing methods mainly embed spatiotemporal knowledge into visual and action representations, and directly perform a cross-modal mapping for step-level action prediction. However, such spatiotemporal reasoning remains largely implicit, making it difficult to handle multiple sequential behaviors with explicit spatiotemporal boundaries. In this work, we propose ST-$\pi$, a structured spatiotemporal VLA model for robotic manipulation. Our model is guided by two key designs: 1) Spatiotemporal VLM. We encode 4D observations and task instructions into latent spaces, and feed them into the LLM to generate a sequence of causally ordered chunk-level action prompts consisting of sub-tasks, spatial grounding and temporal grounding. 2) Spatiotemporal action expert. Conditioned on chunk-level action prompts, we design a structured dual-generator guidance to jointly model spatial dependencies and temporal causality, thus predicting step-level action parameters. Within this structured framework, the VLM explicitly plans global spatiotemporal behavior, and the action expert further refines local spatiotemporal control. In addition, we propose a real-world robotic dataset with structured spatiotemporal annotations for fine-tuning. Extensive experiments have been conducted to demonstrate the effectiveness of our model. Our code link: https://github.com/chuanhaoma/ST-pi.