Neural 4D Evolution under Large Topological Changes from 2D Images

TL;DR

This paper introduces a novel neural framework for capturing 4D shapes with significant topological changes from 2D images, addressing the limitations of existing 3D-based methods and enabling reconstruction of complex dynamic scenes.

Contribution

It proposes new architectures and techniques for 4D shape evolution, including a discretization method, temporal consistency, color rendering, and disentanglement from RGB images, advancing the state-of-the-art in dynamic scene reconstruction.

Findings

Effective reconstruction of 4D shapes with large topological changes

Disentanglement of geometry and appearance from 2D images

Open-source code and dataset for further research

Abstract

In the literature, it has been shown that the evolution of the known explicit 3D surface to the target one can be learned from 2D images using the instantaneous flow field, where the known and target 3D surfaces may largely differ in topology. We are interested in capturing 4D shapes whose topology changes largely over time. We encounter that the straightforward extension of the existing 3D-based method to the desired 4D case performs poorly. In this work, we address the challenges in extending 3D neural evolution to 4D under large topological changes by proposing two novel modifications. More precisely, we introduce (i) a new architecture to discretize and encode the deformation and learn the SDF and (ii) a technique to impose the temporal consistency. (iii) Also, we propose a rendering scheme for color prediction based on Gaussian splatting. Furthermore, to facilitate learning directly from 2D images, we propose a learning framework that can disentangle the geometry and appearance from RGB images. This method of disentanglement, while also useful for the 4D evolution problem that we are concentrating on, is also novel and valid for static scenes. Our extensive experiments on various data provide awesome results and, most importantly, open a new approach toward reconstructing challenging scenes with significant topological changes and deformations. Our source code and the dataset are publicly available at https://github.com/insait-institute/N4DE.