DaRePlane: Direction-aware Representations for Dynamic Scene Reconstruction

TL;DR

DaRePlane introduces a direction-aware representation for dynamic scene reconstruction that captures complex motions efficiently, improving state-of-the-art novel view synthesis in both NeRF and Gaussian splatting frameworks.

Contribution

It proposes a novel direction-aware representation with wavelet transformation and a trainable masking approach, enhancing dynamic scene modeling beyond existing plane-based methods.

Findings

Achieves state-of-the-art results in novel view synthesis.

Effectively models complex scene motions.

Reduces redundancy with trainable masking.

Abstract

Numerous recent approaches to modeling and re-rendering dynamic scenes leverage plane-based explicit representations, addressing slow training times associated with models like neural radiance fields (NeRF) and Gaussian splatting (GS). However, merely decomposing 4D dynamic scenes into multiple 2D plane-based representations is insufficient for high-fidelity re-rendering of scenes with complex motions. In response, we present DaRePlane, a novel direction-aware representation approach that captures scene dynamics from six different directions. This learned representation undergoes an inverse dual-tree complex wavelet transformation (DTCWT) to recover plane-based information. Within NeRF pipelines, DaRePlane computes features for each space-time point by fusing vectors from these recovered planes, then passed to a tiny MLP for color regression. When applied to Gaussian splatting, DaRePlane computes the features of Gaussian points, followed by a tiny multi-head MLP for spatial-time deformation prediction. Notably, to address redundancy introduced by the six real and six imaginary direction-aware wavelet coefficients, we introduce a trainable masking approach, mitigating storage issues without significant performance decline. To demonstrate the generality and efficiency of DaRePlane, we test it on both regular and surgical dynamic scenes, for both NeRF and GS systems. Extensive experiments show that DaRePlane yields state-of-the-art performance in novel view synthesis for various complex dynamic scenes.