4D Primitive-M\^ach\'e: Glueing Primitives for Persistent 4D Scene Reconstruction

TL;DR

This paper introduces a system that reconstructs dynamic 4D scenes from monocular videos by decomposing scenes into moving primitives, enabling persistent, replayable 3D reconstructions over time.

Contribution

It proposes a novel method for joint inference of primitive motions and scene reconstruction, including extrapolation for invisible objects, advancing 4D scene understanding from monocular videos.

Findings

Outperforms existing methods on object scanning datasets

Provides continuous 4D reconstructions with object permanence

Enables replayable 3D scene visualization over time

Abstract

We present a dynamic reconstruction system that receives a casual monocular RGB video as input, and outputs a complete and persistent reconstruction of the scene. In other words, we reconstruct not only the the currently visible parts of the scene, but also all previously viewed parts, which enables replaying the complete reconstruction across all timesteps. Our method decomposes the scene into a set of rigid 3D primitives, which are assumed to be moving throughout the scene. Using estimated dense 2D correspondences, we jointly infer the rigid motion of these primitives through an optimisation pipeline, yielding a 4D reconstruction of the scene, i.e. providing 3D geometry dynamically moving through time. To achieve this, we also introduce a mechanism to extrapolate motion for objects that become invisible, employing motion-grouping techniques to maintain continuity. The resulting system enables 4D spatio-temporal awareness, offering capabilities such as replayable 3D reconstructions of articulated objects through time, multi-object scanning, and object permanence. On object scanning and multi-object datasets, our system significantly outperforms existing methods both quantitatively and qualitatively.