CausalGS: Learning Physical Causality of 3D Dynamic Scenes with Gaussian Representations

TL;DR

CausalGS is a novel framework that learns physical causality in 3D dynamic scenes from multi-view videos without explicit priors, enabling accurate future predictions and view interpolations.

Contribution

It introduces an inverse physics inference module and a differentiable physics simulator to learn scene dynamics solely from videos, without human annotations or strong priors.

Findings

Outperforms state-of-the-art in long-term future frame extrapolation.

Excels in novel view interpolation tasks.

Learns complex physical interactions without human annotations.

Abstract

Learning a physical model from video data that can comprehend physical laws and predict the future trajectories of objects is a formidable challenge in artificial intelligence. Prior approaches either leverage various Partial Differential Equations (PDEs) as soft constraints in the form of PINN losses, or integrate physics simulators into neural networks; however, they often rely on strong priors or high-quality geometry reconstruction. In this paper, we propose CausalGS, a framework that learns the causal dynamics of complex dynamic 3D scenes solely from multi-view videos, while dispensing with the reliance on explicit priors. At its core is an inverse physics inference module that decouples the complex dynamics problem from the video into the joint inference of two factors: the initial velocity field representing the scene's kinematics, and the intrinsic material properties governing its dynamics. This inferred physical information is then utilized within a differentiable physics simulator to guide the learning process in a physics-regularized manner. Extensive experiments demonstrate that CausalGS surpasses the state-of-the-art on the highly challenging task of long-term future frame extrapolation, while also exhibiting advanced performance in novel view interpolation. Crucially, our work shows that, without any human annotation, the model is able to learn the complex interactions between multiple physical properties and understand the causal relationships driving the scene's dynamic evolution, solely from visual observations.