Root Pose Decomposition Towards Generic Non-rigid 3D Reconstruction with Monocular Videos

TL;DR

This paper introduces Root Pose Decomposition (RPD), a novel method for non-rigid 3D reconstruction from monocular videos that handles large deformations, occlusions, and diverse object scales without relying on category-specific templates.

Contribution

The paper proposes RPD, a new approach that decomposes root pose and local transformations, enabling high-fidelity 3D reconstruction of generic objects in complex, real-world scenarios.

Findings

RPD outperforms state-of-the-art methods on DAVIS, OVIS, and AMA datasets.

It effectively handles large deformations and occlusions.

The method scales to diverse object categories in the wild.

Abstract

This work focuses on the 3D reconstruction of non-rigid objects based on monocular RGB video sequences. Concretely, we aim at building high-fidelity models for generic object categories and casually captured scenes. To this end, we do not assume known root poses of objects, and do not utilize category-specific templates or dense pose priors. The key idea of our method, Root Pose Decomposition (RPD), is to maintain a per-frame root pose transformation, meanwhile building a dense field with local transformations to rectify the root pose. The optimization of local transformations is performed by point registration to the canonical space. We also adapt RPD to multi-object scenarios with object occlusions and individual differences. As a result, RPD allows non-rigid 3D reconstruction for complicated scenarios containing objects with large deformations, complex motion patterns, occlusions, and scale diversities of different individuals. Such a pipeline potentially scales to diverse sets of objects in the wild. We experimentally show that RPD surpasses state-of-the-art methods on the challenging DAVIS, OVIS, and AMA datasets.