Manipulating Elasto-Plastic Objects With 3D Occupancy and Learning-Based Predictive Control

TL;DR

This paper introduces a novel framework for manipulating elasto-plastic objects using 3D occupancy representation, a learned dynamics model, and a learning-based predictive control, enabling effective shaping of complex deformable objects in simulation and real-world tests.

Contribution

It presents a new approach combining 3D occupancy, deep neural networks, and predictive control for elasto-plastic object manipulation, addressing challenges of self-occlusion and complex dynamics.

Findings

Successfully shapes objects into target configurations in simulation.

Demonstrates real-world applicability with effective manipulation.

Improves planning efficiency with a shape-based action initialization.

Abstract

Manipulating elasto-plastic objects remains a significant challenge due to severe self-occlusion, difficulties of representation, and complicated dynamics. This work proposes a novel framework for elasto-plastic object manipulation with a quasi-static assumption for motions, leveraging 3D occupancy to represent such objects, a learned dynamics model trained with 3D occupancy, and a learning-based predictive control algorithm to address these challenges effectively. We build a novel data collection platform to collect full spatial information and propose a pipeline for generating a 3D occupancy dataset. To infer the 3D occupancy during manipulation, an occupancy prediction network is trained with multiple RGB images supervised by the generated dataset. We design a deep neural network empowered by a 3D convolution neural network (CNN) and a graph neural network (GNN) to predict the complex deformation with the inferred 3D occupancy results. A learning-based predictive control algorithm is introduced to plan the robot actions, incorporating a novel shape-based action initialization module specifically designed to improve the planner efficiency. The proposed framework in this paper can successfully shape the elasto-plastic objects into a given goal shape and has been verified in various experiments both in simulation and the real world.