DynMF: Neural Motion Factorization for Real-time Dynamic View Synthesis with 3D Gaussian Splatting

TL;DR

DynMF introduces a neural motion factorization method that enables real-time, high-quality dynamic scene rendering by decomposing motions into learned trajectories, achieving fast training and synthesis with interpretability and control.

Contribution

The paper presents DynMF, a novel neural representation that decomposes dynamic scenes into a small set of learned trajectories, enabling real-time view synthesis with high quality and interpretability.

Findings

Achieves 120 FPS rendering speed comparable to 3D Gaussian Splatting.

Reaches state-of-the-art quality within 5 minutes of training.

Allows disentanglement and control of scene motions for novel motion synthesis.

Abstract

Accurately and efficiently modeling dynamic scenes and motions is considered so challenging a task due to temporal dynamics and motion complexity. To address these challenges, we propose DynMF, a compact and efficient representation that decomposes a dynamic scene into a few neural trajectories. We argue that the per-point motions of a dynamic scene can be decomposed into a small set of explicit or learned trajectories. Our carefully designed neural framework consisting of a tiny set of learned basis queried only in time allows for rendering speed similar to 3D Gaussian Splatting, surpassing 120 FPS, while at the same time, requiring only double the storage compared to static scenes. Our neural representation adequately constrains the inherently underconstrained motion field of a dynamic scene leading to effective and fast optimization. This is done by biding each point to motion coefficients that enforce the per-point sharing of basis trajectories. By carefully applying a sparsity loss to the motion coefficients, we are able to disentangle the motions that comprise the scene, independently control them, and generate novel motion combinations that have never been seen before. We can reach state-of-the-art render quality within just 5 minutes of training and in less than half an hour, we can synthesize novel views of dynamic scenes with superior photorealistic quality. Our representation is interpretable, efficient, and expressive enough to offer real-time view synthesis of complex dynamic scene motions, in monocular and multi-view scenarios.