Particle-Grid Neural Dynamics for Learning Deformable Object Models from RGB-D Videos

TL;DR

This paper introduces a neural dynamics framework combining particles and spatial grids to learn and simulate the complex behavior of deformable objects from RGB-D videos, enabling realistic modeling and manipulation.

Contribution

It presents a novel hybrid particle-grid neural model that captures deformable object dynamics and generalizes across object categories from sparse visual data.

Findings

Outperforms existing simulators in limited-view scenarios

Successfully models diverse deformable objects like ropes and cloths

Enables goal-conditioned manipulation using learned dynamics

Abstract

Modeling the dynamics of deformable objects is challenging due to their diverse physical properties and the difficulty of estimating states from limited visual information. We address these challenges with a neural dynamics framework that combines object particles and spatial grids in a hybrid representation. Our particle-grid model captures global shape and motion information while predicting dense particle movements, enabling the modeling of objects with varied shapes and materials. Particles represent object shapes, while the spatial grid discretizes the 3D space to ensure spatial continuity and enhance learning efficiency. Coupled with Gaussian Splattings for visual rendering, our framework achieves a fully learning-based digital twin of deformable objects and generates 3D action-conditioned videos. Through experiments, we demonstrate that our model learns the dynamics of diverse objects -- such as ropes, cloths, stuffed animals, and paper bags -- from sparse-view RGB-D recordings of robot-object interactions, while also generalizing at the category level to unseen instances. Our approach outperforms state-of-the-art learning-based and physics-based simulators, particularly in scenarios with limited camera views. Furthermore, we showcase the utility of our learned models in model-based planning, enabling goal-conditioned object manipulation across a range of tasks. The project page is available at https://kywind.github.io/pgnd .