Embodiment-Agnostic Action Planning via Object-Part Scene Flow

TL;DR

This paper introduces an embodiment-agnostic action planning method that predicts 3D object-part scene flow to generate robust, generalizable robot action trajectories from human demonstrations, applicable across diverse embodiments.

Contribution

It presents a novel approach combining object-part scene flow prediction with embodiment-agnostic trajectory planning, enabling cross-embodiment generalization and learning from human demonstrations.

Findings

Outperforms existing methods by 27.7% and 26.2% in virtual environments.

Demonstrates successful real-world deployment across various robot embodiments.

Achieves robust action planning without requiring trajectory data during training.

Abstract

Observing that the key for robotic action planning is to understand the target-object motion when its associated part is manipulated by the end effector, we propose to generate the 3D object-part scene flow and extract its transformations to solve the action trajectories for diverse embodiments. The advantage of our approach is that it derives the robot action explicitly from object motion prediction, yielding a more robust policy by understanding the object motions. Also, beyond policies trained on embodiment-centric data, our method is embodiment-agnostic, generalizable across diverse embodiments, and being able to learn from human demonstrations. Our method comprises three components: an object-part predictor to locate the part for the end effector to manipulate, an RGBD video generator to predict future RGBD videos, and a trajectory planner to extract embodiment-agnostic transformation sequences and solve the trajectory for diverse embodiments. Trained on videos even without trajectory data, our method still outperforms existing works significantly by 27.7% and 26.2% on the prevailing virtual environments MetaWorld and Franka-Kitchen, respectively. Furthermore, we conducted real-world experiments, showing that our policy, trained only with human demonstration, can be deployed to various embodiments.