Dynamic Gaussian Marbles for Novel View Synthesis of Casual Monocular Videos

TL;DR

This paper introduces Dynamic Gaussian Marbles, a novel method for novel view synthesis from monocular videos that overcomes the limitations of existing Gaussian-based methods by incorporating priors, hierarchical learning, and simplified Gaussian shapes.

Contribution

It extends Gaussian scene representations to casual monocular videos through three key modifications: isotropic Gaussian marbles, hierarchical learning, and priors including tracking, enabling effective 3D scene capture.

Findings

Outperforms Gaussian baselines in quality on dynamic scenes

Achieves comparable results to non-Gaussian methods

Maintains efficiency, editability, and tracking benefits

Abstract

Gaussian splatting has become a popular representation for novel-view synthesis, exhibiting clear strengths in efficiency, photometric quality, and compositional edibility. Following its success, many works have extended Gaussians to 4D, showing that dynamic Gaussians maintain these benefits while also tracking scene geometry far better than alternative representations. Yet, these methods assume dense multi-view videos as supervision. In this work, we are interested in extending the capability of Gaussian scene representations to casually captured monocular videos. We show that existing 4D Gaussian methods dramatically fail in this setup because the monocular setting is underconstrained. Building off this finding, we propose a method we call Dynamic Gaussian Marbles, which consist of three core modifications that target the difficulties of the monocular setting. First, we use isotropic Gaussian "marbles'', reducing the degrees of freedom of each Gaussian. Second, we employ a hierarchical divide and-conquer learning strategy to efficiently guide the optimization towards solutions with globally coherent motion. Finally, we add image-level and geometry-level priors into the optimization, including a tracking loss that takes advantage of recent progress in point tracking. By constraining the optimization, Dynamic Gaussian Marbles learns Gaussian trajectories that enable novel-view rendering and accurately capture the 3D motion of the scene elements. We evaluate on the Nvidia Dynamic Scenes dataset and the DyCheck iPhone dataset, and show that Gaussian Marbles significantly outperforms other Gaussian baselines in quality, and is on-par with non-Gaussian representations, all while maintaining the efficiency, compositionality, editability, and tracking benefits of Gaussians. Our project page can be found here https://geometry.stanford.edu/projects/dynamic-gaussian-marbles.github.io/.