M-PhyGs: Multi-Material Object Dynamics from Video

TL;DR

This paper presents M-PhyGs, a novel method for estimating the complex multi-material composition and physical parameters of natural objects like flowers from video, advancing the understanding of their dynamics in real-world settings.

Contribution

Introduction of M-PhyGs, a method that jointly segments multi-material objects and estimates their physical parameters from video, handling complex geometries and compositions.

Findings

M-PhyGs accurately estimates material composition and parameters from video.

The method effectively segments multi-material objects in natural scenes.

Experimental results demonstrate the approach's robustness and precision.

Abstract

Knowledge of the physical material properties governing the dynamics of a real-world object becomes necessary to accurately anticipate its response to unseen interactions. Existing methods for estimating such physical material parameters from visual data assume homogeneous single-material objects, pre-learned dynamics, or simplistic topologies. Real-world objects, however, are often complex in material composition and geometry lying outside the realm of these assumptions. In this paper, we particularly focus on flowers as a representative common object. We introduce Multi-material Physical Gaussians (M-PhyGs) to estimate the material composition and parameters of such multi-material complex natural objects from video. From a short video captured in a natural setting, M-PhyGs jointly segments the object into similar materials and recovers their continuum mechanical parameters while accounting for gravity. M-PhyGs achieves this efficiently with newly introduced cascaded 3D and 2D losses, and by leveraging temporal mini-batching. We introduce a dataset, Phlowers, of people interacting with flowers as a novel platform to evaluate the accuracy of this challenging task of multi-material physical parameter estimation. Experimental results on Phlowers dataset demonstrate the accuracy and effectiveness of M-PhyGs and its components.