Monocular Dynamic Gaussian Splatting: Fast, Brittle, and Scene Complexity Rules

TL;DR

This paper benchmarks and analyzes monocular dynamic Gaussian splatting methods, revealing their strengths and limitations, especially regarding scene complexity and optimization brittleness, to guide future research in dynamic scene view synthesis.

Contribution

It provides the first comprehensive benchmark and analysis of Gaussian splatting methods for monocular dynamic scene reconstruction, including a new synthetic dataset.

Findings

Synthetic data shows clear performance ranking among methods.

Real-world data complexity diminishes differences between methods.

Fast rendering speeds lead to optimization brittleness.

Abstract

Gaussian splatting methods are emerging as a popular approach for converting multi-view image data into scene representations that allow view synthesis. In particular, there is interest in enabling view synthesis for dynamic scenes using only monocular input data -- an ill-posed and challenging problem. The fast pace of work in this area has produced multiple simultaneous papers that claim to work best, which cannot all be true. In this work, we organize, benchmark, and analyze many Gaussian-splatting-based methods, providing apples-to-apples comparisons that prior works have lacked. We use multiple existing datasets and a new instructive synthetic dataset designed to isolate factors that affect reconstruction quality. We systematically categorize Gaussian splatting methods into specific motion representation types and quantify how their differences impact performance. Empirically, we find that their rank order is well-defined in synthetic data, but the complexity of real-world data currently overwhelms the differences. Furthermore, the fast rendering speed of all Gaussian-based methods comes at the cost of brittleness in optimization. We summarize our experiments into a list of findings that can help to further progress in this lively problem setting.