Class-agnostic Reconstruction of Dynamic Objects from Videos

TL;DR

REDO is a unified, class-agnostic framework that reconstructs complete dynamic object shapes from videos, handling occlusion, various motions, and multiple categories using novel 4D implicit and transformation modules.

Contribution

The paper introduces REDO, a novel framework with two modules for class-agnostic dynamic object reconstruction from videos, addressing occlusion, diverse motions, and multiple categories.

Findings

REDO outperforms state-of-the-art methods on synthetic and real-world datasets.

The canonical 4D implicit function effectively captures object shape and appearance.

The 4D transformation module accurately models object dynamics.

Abstract

We introduce REDO, a class-agnostic framework to REconstruct the Dynamic Objects from RGBD or calibrated videos. Compared to prior work, our problem setting is more realistic yet more challenging for three reasons: 1) due to occlusion or camera settings an object of interest may never be entirely visible, but we aim to reconstruct the complete shape; 2) we aim to handle different object dynamics including rigid motion, non-rigid motion, and articulation; 3) we aim to reconstruct different categories of objects with one unified framework. To address these challenges, we develop two novel modules. First, we introduce a canonical 4D implicit function which is pixel-aligned with aggregated temporal visual cues. Second, we develop a 4D transformation module which captures object dynamics to support temporal propagation and aggregation. We study the efficacy of REDO in extensive experiments on synthetic RGBD video datasets SAIL-VOS 3D and DeformingThings4D++, and on real-world video data 3DPW. We find REDO outperforms state-of-the-art dynamic reconstruction methods by a margin. In ablation studies we validate each developed component.