DoughNet: A Visual Predictive Model for Topological Manipulation of Deformable Objects

TL;DR

DoughNet is a Transformer-based model that predicts topological and geometrical changes in deformable objects like dough, enabling better planning of robotic manipulation involving splitting and merging.

Contribution

Introducing DoughNet, a novel autoencoder and autoregressive set prediction model for topological manipulation of deformable objects in latent space.

Findings

Outperforms existing deformation prediction methods

Accurately predicts topological changes in simulated environments

Effective in real-world robotic manipulation tasks

Abstract

Manipulation of elastoplastic objects like dough often involves topological changes such as splitting and merging. The ability to accurately predict these topological changes that a specific action might incur is critical for planning interactions with elastoplastic objects. We present DoughNet, a Transformer-based architecture for handling these challenges, consisting of two components. First, a denoising autoencoder represents deformable objects of varying topology as sets of latent codes. Second, a visual predictive model performs autoregressive set prediction to determine long-horizon geometrical deformation and topological changes purely in latent space. Given a partial initial state and desired manipulation trajectories, it infers all resulting object geometries and topologies at each step. DoughNet thereby allows to plan robotic manipulation; selecting a suited tool, its pose and opening width to recreate robot- or human-made goals. Our experiments in simulated and real environments show that DoughNet is able to significantly outperform related approaches that consider deformation only as geometrical change.