DexSim2Real$^{2}$: Building Explicit World Model for Precise Articulated Object Dexterous Manipulation

TL;DR

DexSim2Real2 introduces an explicit world model for goal-conditioned articulated object manipulation, enabling precise control without demonstrations or reinforcement learning by combining active interactions, 3D AIGC modeling, and eigengrasp for efficient planning.

Contribution

The paper presents a novel framework that constructs an explicit world model for unseen articulated objects using active interactions and 3D AIGC, facilitating precise manipulation without RL or demonstrations.

Findings

Effective manipulation with suction and gripper tools.

Generalizes to tool-based manipulation strategies.

Validates precise control on various robotic setups.

Abstract

Articulated objects are ubiquitous in daily life. In this paper, we present DexSim2Real$^{2}$, a novel framework for goal-conditioned articulated object manipulation. The core of our framework is constructing an explicit world model of unseen articulated objects through active interactions, which enables sampling-based model predictive control to plan trajectories achieving different goals without requiring demonstrations or RL. It first predicts an interaction using an affordance network trained on self-supervised interaction data or videos of human manipulation. After executing the interactions on the real robot to move the object parts, we propose a novel modeling pipeline based on 3D AIGC to build a digital twin of the object in simulation from multiple frames of observations. For dexterous hands, we utilize eigengrasp to reduce the action dimension, enabling more efficient trajectory searching. Experiments validate the framework's effectiveness for precise manipulation using a suction gripper, a two-finger gripper and two dexterous hand. The generalizability of the explicit world model also enables advanced manipulation strategies like manipulating with tools.