A Bayesian Treatment of Real-to-Sim for Deformable Object Manipulation

TL;DR

This paper introduces a probabilistic Bayesian approach to infer physical parameters of deformable objects from image sequences, enabling better simulation and manipulation in robotics.

Contribution

It presents a novel method for representing deformable object states as distribution embeddings and integrates noisy observations into Bayesian inference for parameter estimation.

Findings

Successfully estimates posterior distributions of physical properties

Handles noisy state observations effectively

Applies to complex deformable objects like cloth and ropes

Abstract

Deformable object manipulation remains a challenging task in robotics research. Conventional techniques for parameter inference and state estimation typically rely on a precise definition of the state space and its dynamics. While this is appropriate for rigid objects and robot states, it is challenging to define the state space of a deformable object and how it evolves in time. In this work, we pose the problem of inferring physical parameters of deformable objects as a probabilistic inference task defined with a simulator. We propose a novel methodology for extracting state information from image sequences via a technique to represent the state of a deformable object as a distribution embedding. This allows to incorporate noisy state observations directly into modern Bayesian simulation-based inference tools in a principled manner. Our experiments confirm that we can estimate posterior distributions of physical properties, such as elasticity, friction and scale of highly deformable objects, such as cloth and ropes. Overall, our method addresses the real-to-sim problem probabilistically and helps to better represent the evolution of the state of deformable objects.