Neural Material Adaptor for Visual Grounding of Intrinsic Dynamics

TL;DR

This paper introduces NeuMA, a neural material adaptor that combines physical laws with learned corrections for accurate, generalizable, and interpretable visual grounding of intrinsic dynamics, supported by a novel particle-driven 3D Gaussian Splatting method.

Contribution

The paper presents NeuMA, a novel framework integrating physical laws with learned corrections, and Particle-GS, a new particle-driven 3D Gaussian Splatting technique for dynamic scene understanding.

Findings

NeuMA accurately captures intrinsic dynamics across various scenarios.

Particle-GS effectively bridges simulation and observed images.

The approach outperforms existing methods in dynamic rendering and generalization.

Abstract

While humans effortlessly discern intrinsic dynamics and adapt to new scenarios, modern AI systems often struggle. Current methods for visual grounding of dynamics either use pure neural-network-based simulators (black box), which may violate physical laws, or traditional physical simulators (white box), which rely on expert-defined equations that may not fully capture actual dynamics. We propose the Neural Material Adaptor (NeuMA), which integrates existing physical laws with learned corrections, facilitating accurate learning of actual dynamics while maintaining the generalizability and interpretability of physical priors. Additionally, we propose Particle-GS, a particle-driven 3D Gaussian Splatting variant that bridges simulation and observed images, allowing back-propagate image gradients to optimize the simulator. Comprehensive experiments on various dynamics in terms of grounded particle accuracy, dynamic rendering quality, and generalization ability demonstrate that NeuMA can accurately capture intrinsic dynamics.