3D Dynamics-Aware Manipulation: Endowing Manipulation Policies with 3D Foresight

TL;DR

This paper introduces a 3D dynamics-aware manipulation framework that enhances manipulation policies with 3D foresight through self-supervised learning tasks, improving robustness in depth-wise movements.

Contribution

It presents a novel integration of 3D world modeling with policy learning, utilizing self-supervised tasks to improve manipulation performance in 3D environments.

Findings

Significant performance boost in manipulation tasks with 3D foresight

Effective in both simulation and real-world scenarios

Maintains inference speed despite added complexity

Abstract

The incorporation of world modeling into manipulation policy learning has pushed the boundary of manipulation performance. However, existing efforts simply model the 2D visual dynamics, which is insufficient for robust manipulation when target tasks involve prominent depth-wise movement. To address this, we present a 3D dynamics-aware manipulation framework that seamlessly integrates 3D world modeling and policy learning. Three self-supervised learning tasks (current depth estimation, future RGB-D prediction, 3D flow prediction) are introduced within our framework, which complement each other and endow the policy model with 3D foresight. Extensive experiments on simulation and the real world show that 3D foresight can greatly boost the performance of manipulation policies without sacrificing inference speed. Code is available at https://github.com/Stardust-hyx/3D-Foresight.