NVFi: Neural Velocity Fields for 3D Physics Learning from Dynamic Videos

TL;DR

NVFi introduces a novel approach to model 3D scene dynamics from multi-view videos, enabling applications like future frame prediction and 3D scene understanding without supervision.

Contribution

The paper presents NVFi, a method that jointly learns geometry, appearance, and velocity of 3D scenes from videos, with new datasets and superior performance in dynamic scene tasks.

Findings

Outperforms baseline methods in future frame extrapolation

Enables unsupervised 3D semantic scene decomposition

Effective in modeling complex 3D scene dynamics

Abstract

In this paper, we aim to model 3D scene dynamics from multi-view videos. Unlike the majority of existing works which usually focus on the common task of novel view synthesis within the training time period, we propose to simultaneously learn the geometry, appearance, and physical velocity of 3D scenes only from video frames, such that multiple desirable applications can be supported, including future frame extrapolation, unsupervised 3D semantic scene decomposition, and dynamic motion transfer. Our method consists of three major components, 1) the keyframe dynamic radiance field, 2) the interframe velocity field, and 3) a joint keyframe and interframe optimization module which is the core of our framework to effectively train both networks. To validate our method, we further introduce two dynamic 3D datasets: 1) Dynamic Object dataset, and 2) Dynamic Indoor Scene dataset. We conduct extensive experiments on multiple datasets, demonstrating the superior performance of our method over all baselines, particularly in the critical tasks of future frame extrapolation and unsupervised 3D semantic scene decomposition.