Unbiased 4D: Monocular 4D Reconstruction with a Neural Deformation Model

TL;DR

This paper introduces Ub4D, a neural volume rendering approach for monocular 4D scene reconstruction that effectively handles large deformations, occlusions, and shape completion, advancing the state of the art.

Contribution

It presents a novel neural deformation model with an unbiased volume rendering formulation and a dynamic scene flow loss for improved monocular 4D reconstruction.

Findings

Outperforms existing methods in surface reconstruction accuracy

Demonstrates robustness to large scene deformations

Operates effectively on monocular RGB videos

Abstract

Capturing general deforming scenes from monocular RGB video is crucial for many computer graphics and vision applications. However, current approaches suffer from drawbacks such as struggling with large scene deformations, inaccurate shape completion or requiring 2D point tracks. In contrast, our method, Ub4D, handles large deformations, performs shape completion in occluded regions, and can operate on monocular RGB videos directly by using differentiable volume rendering. This technique includes three new in the context of non-rigid 3D reconstruction components, i.e., 1) A coordinate-based and implicit neural representation for non-rigid scenes, which in conjunction with differentiable volume rendering enables an unbiased reconstruction of dynamic scenes, 2) a proof that extends the unbiased formulation of volume rendering to dynamic scenes, and 3) a novel dynamic scene flow loss, which enables the reconstruction of larger deformations by leveraging the coarse estimates of other methods. Results on our new dataset, which will be made publicly available, demonstrate a clear improvement over the state of the art in terms of surface reconstruction accuracy and robustness to large deformations.