Learning Latent Graph Dynamics for Visual Manipulation of Deformable Objects

TL;DR

This paper introduces G-DOOM, a novel approach that learns latent graph dynamics from visual data to enable effective manipulation of deformable objects like ropes and cloth in robotics.

Contribution

G-DOOM automatically extracts keypoints and learns a graph neural network to model deformable object dynamics from images, improving manipulation planning and transfer to real robots.

Findings

Strong performance on challenging rope and cloth tasks

Successful transfer from simulation to real robot manipulation

Outperforms state-of-the-art methods in experiments

Abstract

Manipulating deformable objects, such as ropes and clothing, is a long-standing challenge in robotics, because of their large degrees of freedom, complex non-linear dynamics, and self-occlusion in visual perception. The key difficulty is a suitable representation, rich enough to capture the object shape, dynamics for manipulation and yet simple enough to be estimated reliably from visual observations. This work aims to learn latent Graph dynamics for DefOrmable Object Manipulation (G-DOOM). G-DOOM approximates a deformable object as a sparse set of interacting keypoints, which are extracted automatically from images via unsupervised learning. It learns a graph neural network that captures abstractly the geometry and the interaction dynamics of the keypoints. To handle object self-occlusion, G-DOOM uses a recurrent neural network to track the keypoints over time and condition their interactions on the history. We then train the resulting recurrent graph dynamics model through contrastive learning in a high-fidelity simulator. For manipulation planning, G-DOOM reasons explicitly about the learned dynamics model through model-predictive control applied at each keypoint. Preliminary experiments of G-DOOM on a set of challenging rope and cloth manipulation tasks indicate strong performance, compared with state-of-the-art methods. Although trained in a simulator, G-DOOM transfers directly to a real robot for both rope and cloth manipulation.