4D Gaussian Splatting: Modeling Dynamic Scenes with Native 4D Primitives

TL;DR

This paper introduces 4D Gaussian Splatting, a novel method for real-time, photorealistic rendering of complex dynamic scenes by modeling them as collections of 4D primitives, enabling efficient view synthesis and scene understanding.

Contribution

It presents the first real-time, end-to-end optimized 4D scene representation using native 4D Gaussians, improving visual quality and efficiency for dynamic scene rendering.

Findings

Achieves real-time rendering of high-resolution dynamic scenes

Outperforms existing methods in visual quality and efficiency

Effectively reduces memory footprint with compact variants

Abstract

Dynamic 3D scene representation and novel view synthesis are crucial for enabling immersive experiences required by AR/VR and metaverse applications. It is a challenging task due to the complexity of unconstrained real-world scenes and their temporal dynamics. In this paper, we reformulate the reconstruction of a time-varying 3D scene as approximating its underlying spatiotemporal 4D volume by optimizing a collection of native 4D primitives, i.e., 4D Gaussians, with explicit geometry and appearance modeling. Equipped with a tailored rendering pipeline, our representation can be end-to-end optimized using only photometric supervision while free viewpoint viewing at interactive frame rate, making it suitable for representing real world scene with complex dynamic. This approach has been the first solution to achieve real-time rendering of high-resolution, photorealistic novel views for complex dynamic scenes. To facilitate real-world applications, we derive several compact variants that effectively reduce the memory footprint to address its storage bottleneck. Extensive experiments validate the superiority of 4DGS in terms of visual quality and efficiency across a range of dynamic scene-related tasks (e.g., novel view synthesis, 4D generation, scene understanding) and scenarios (e.g., single object, indoor scenes, driving environments, synthetic and real data).