Event-boosted Deformable 3D Gaussians for Dynamic Scene Reconstruction

TL;DR

This paper introduces a novel method combining event cameras with deformable 3D Gaussian Splatting for high-temporal-resolution dynamic scene reconstruction, along with a new benchmark dataset.

Contribution

It proposes a joint modeling strategy for Gaussian thresholds, a dynamic-static decomposition approach, and presents the first event-inclusive 4D benchmark for dynamic scenes.

Findings

Achieves state-of-the-art reconstruction quality on the new benchmark.

Effectively separates dynamic and static scene components.

Accelerates rendering by focusing on dynamic regions.

Abstract

Deformable 3D Gaussian Splatting (3D-GS) is limited by missing intermediate motion information due to the low temporal resolution of RGB cameras. To address this, we introduce the first approach combining event cameras, which capture high-temporal-resolution, continuous motion data, with deformable 3D-GS for dynamic scene reconstruction. We observe that threshold modeling for events plays a crucial role in achieving high-quality reconstruction. Therefore, we propose a GS-Threshold Joint Modeling strategy, creating a mutually reinforcing process that greatly improves both 3D reconstruction and threshold modeling. Moreover, we introduce a Dynamic-Static Decomposition strategy that first identifies dynamic areas by exploiting the inability of static Gaussians to represent motions, then applies a buffer-based soft decomposition to separate dynamic and static areas. This strategy accelerates rendering by avoiding unnecessary deformation in static areas, and focuses on dynamic areas to enhance fidelity. Additionally, we contribute the first event-inclusive 4D benchmark with synthetic and real-world dynamic scenes, on which our method achieves state-of-the-art performance.