Dynamic 3D Gaussian Fields for Urban Areas

TL;DR

This paper introduces 4DGF, a neural scene representation that enables fast, high-quality, and scalable novel-view synthesis for large, dynamic urban environments, effectively handling heterogeneous data and scene dynamics.

Contribution

The paper presents 4DGF, a novel neural scene representation combining 3D Gaussians and neural fields for scalable, dynamic urban scene modeling with improved rendering speed and quality.

Findings

Surpasses state-of-the-art by over 3 dB in PSNR.

Achieves more than 200 times faster rendering speeds.

Effectively handles heterogeneous data and complex scene dynamics.

Abstract

We present an efficient neural 3D scene representation for novel-view synthesis (NVS) in large-scale, dynamic urban areas. Existing works are not well suited for applications like mixed-reality or closed-loop simulation due to their limited visual quality and non-interactive rendering speeds. Recently, rasterization-based approaches have achieved high-quality NVS at impressive speeds. However, these methods are limited to small-scale, homogeneous data, i.e. they cannot handle severe appearance and geometry variations due to weather, season, and lighting and do not scale to larger, dynamic areas with thousands of images. We propose 4DGF, a neural scene representation that scales to large-scale dynamic urban areas, handles heterogeneous input data, and substantially improves rendering speeds. We use 3D Gaussians as an efficient geometry scaffold while relying on neural fields as a compact and flexible appearance model. We integrate scene dynamics via a scene graph at global scale while modeling articulated motions on a local level via deformations. This decomposed approach enables flexible scene composition suitable for real-world applications. In experiments, we surpass the state-of-the-art by over 3 dB in PSNR and more than 200 times in rendering speed.