DGD: Dynamic 3D Gaussians Distillation

TL;DR

This paper introduces DGD, a novel method for learning dynamic 3D semantic radiance fields from a single monocular video, enabling high-quality, fast rendering of novel views with semantic segmentation and tracking.

Contribution

DGD presents a unified 3D representation that jointly optimizes appearance and semantic attributes over time, improving dynamic scene understanding from monocular videos.

Findings

Enables dense semantic 3D object tracking.

Produces high-quality, fast rendering results.

Works effectively across diverse dynamic scenes.

Abstract

We tackle the task of learning dynamic 3D semantic radiance fields given a single monocular video as input. Our learned semantic radiance field captures per-point semantics as well as color and geometric properties for a dynamic 3D scene, enabling the generation of novel views and their corresponding semantics. This enables the segmentation and tracking of a diverse set of 3D semantic entities, specified using a simple and intuitive interface that includes a user click or a text prompt. To this end, we present DGD, a unified 3D representation for both the appearance and semantics of a dynamic 3D scene, building upon the recently proposed dynamic 3D Gaussians representation. Our representation is optimized over time with both color and semantic information. Key to our method is the joint optimization of the appearance and semantic attributes, which jointly affect the geometric properties of the scene. We evaluate our approach in its ability to enable dense semantic 3D object tracking and demonstrate high-quality results that are fast to render, for a diverse set of scenes. Our project webpage is available on https://isaaclabe.github.io/DGD-Website/