Rapid Adaptation of Particle Dynamics for Generalized Deformable Object Mobile Manipulation

TL;DR

This paper introduces RAPiD, a method that extends rapid motor adaptation to deformable object manipulation by encoding shape-changing dynamics from particle positions, enabling effective real-world manipulation with high success rates.

Contribution

The paper presents RAPiD, a novel two-phase approach that learns to manipulate deformable objects by inferring shape-changing dynamics from visual data, bridging simulation and real-world application.

Findings

Achieved over 80% success rate in real-world deformable object manipulation tasks.

Effectively transferred policies from simulation to real robot using visual observations.

Demonstrated robustness across various object types and dynamic conditions.

Abstract

We address the challenge of learning to manipulate deformable objects with unknown dynamics. In non-rigid objects, the dynamics parameters define how they react to interactions -- how they stretch, bend, compress, and move -- and they are critical to determining the optimal actions to perform a manipulation task successfully. In other robotic domains, such as legged locomotion and in-hand rigid object manipulation, state-of-the-art approaches can handle unknown dynamics using Rapid Motor Adaptation (RMA). Through a supervised procedure in simulation that encodes each rigid object's dynamics, such as mass and position, these approaches learn a policy that conditions actions on a vector of latent dynamic parameters inferred from sequences of state-actions. However, in deformable object manipulation, the object's dynamics not only includes its mass and position, but also how the shape of the object changes. Our key insight is that the recent ground-truth particle positions of a deformable object in simulation capture changes in the object's shape, making it possible to extend RMA to deformable object manipulation. This key insight allows us to develop RAPiD, a two-phase method that learns to perform real-robot deformable object mobile manipulation by: 1) learning a visuomotor policy conditioned on the object's dynamics embedding, which is encoded from the object's privileged information in simulation, such as its mass and ground-truth particle positions, and 2) learning to infer this embedding using non-privileged information instead, such as robot visual observations and actions, so that the learned policy can transfer to the real world. On a mobile manipulator with 22 degrees of freedom, RAPiD enables over 80%+ success rates across two vision-based deformable object mobile manipulation tasks in the real world, under various object dynamics, categories, and instances.