EMPM: Embodied MPM for Modeling and Simulation of Deformable Objects

TL;DR

EMPM introduces a physics-based deformable object modeling framework using a differentiable MPM simulator, enabling accurate reconstruction, simulation, and adaptive manipulation of complex materials from multi-view RGB-D data.

Contribution

It presents a novel embodied MPM framework that combines geometry reconstruction with physics-based simulation and online parameter optimization for deformable objects.

Findings

Outperforms spring-mass baseline models in experiments.

Enables adaptive, robust, and physics-aware object representations.

Facilitates robotic manipulation of complex deformables.

Abstract

Modeling deformable objects - especially continuum materials - in a way that is physically plausible, generalizable, and data-efficient remains challenging across 3D vision, graphics, and robotic manipulation. Many existing methods oversimplify the rich dynamics of deformable objects or require large training sets, which often limits generalization. We introduce embodied MPM (EMPM), a deformable object modeling and simulation framework built on a differentiable Material Point Method (MPM) simulator that captures the dynamics of challenging materials. From multi-view RGB-D videos, our approach reconstructs geometry and appearance, then uses an MPM physics engine to simulate object behavior by minimizing the mismatch between predicted and observed visual data. We further optimize MPM parameters online using sensory feedback, enabling adaptive, robust, and physics-aware object representations that open new possibilities for robotic manipulation of complex deformables. Experiments show that EMPM outperforms spring-mass baseline models. Project website: https://embodied-mpm.github.io.