Shape of Motion: 4D Reconstruction from a Single Video

TL;DR

This paper presents a novel method for 4D reconstruction of dynamic scenes from a single monocular video, explicitly modeling 3D motion trajectories using low-dimensional motion bases and integrating data-driven priors for improved accuracy.

Contribution

It introduces a new approach that explicitly models 3D motion trajectories with SE(3) bases and consolidates noisy priors for globally consistent dynamic scene reconstruction.

Findings

Achieves state-of-the-art results in 3D/2D motion estimation

Effective long-range motion tracking in dynamic scenes

Improves novel view synthesis quality

Abstract

Monocular dynamic reconstruction is a challenging and long-standing vision problem due to the highly ill-posed nature of the task. Existing approaches depend on templates, are effective only in quasi-static scenes, or fail to model 3D motion explicitly. We introduce a method for reconstructing generic dynamic scenes, featuring explicit, persistent 3D motion trajectories in the world coordinate frame, from casually captured monocular videos. We tackle the problem with two key insights: First, we exploit the low-dimensional structure of 3D motion by representing scene motion with a compact set of SE(3) motion bases. Each point's motion is expressed as a linear combination of these bases, facilitating soft decomposition of the scene into multiple rigidly-moving groups. Second, we take advantage of off-the-shelf data-driven priors such as monocular depth maps and long-range 2D tracks, and devise a method to effectively consolidate these noisy supervisory signals, resulting in a globally consistent representation of the dynamic scene. Experiments show that our method achieves state-of-the-art performance for both long-range 3D/2D motion estimation and novel view synthesis on dynamic scenes. Project Page: https://shape-of-motion.github.io/