Physical Autoregressive Model for Robotic Manipulation without Action Pretraining

TL;DR

This paper introduces a Physical Autoregressive Model (PAR) that leverages video pretraining to understand physical dynamics in robotic manipulation without needing action pretraining, achieving high success rates and accurate predictions.

Contribution

The paper presents a novel PAR model that combines frames and actions as physical tokens, utilizing video pretraining for manipulation tasks without action pretraining, and introduces a DiT-based de-tokenizer and efficiency improvements.

Findings

Achieves 100% success on PushCube in ManiSkill benchmark.

Matches action-pretrained baselines on other tasks.

Accurately predicts future videos with aligned action trajectories.

Abstract

The scarcity of manipulation data has motivated the use of pretrained large models from other modalities in robotics. In this work, we build upon autoregressive video generation models to propose a Physical Autoregressive Model (PAR), where physical tokens combine frames and actions to represent the joint evolution of the robot and its environment. PAR leverages the world knowledge embedded in video pretraining to understand physical dynamics without requiring action pretraining, enabling accurate video prediction and consistent action trajectories. It also adopts a DiT-based de-tokenizer to model frames and actions as continuous tokens, mitigating quantization errors and facilitating mutual enhancement. Furthermore, we incorporate a causal mask with inverse kinematics, parallel training, and the KV-cache mechanism to further improve performance and efficiency. Experiments on the ManiSkill benchmark show that PAR achieves a 100\% success rate on the PushCube task, matches the performance of action-pretrained baselines on other tasks, and accurately predicts future videos with tightly aligned action trajectories. These findings underscore a promising direction for robotic manipulation by transferring world knowledge from autoregressive video pretraining. The project page is here: https://hcplab-sysu.github.io/PhysicalAutoregressiveModel/