Learning monocular 3D reconstruction of articulated categories from motion

TL;DR

This paper presents a novel self-supervised approach for monocular 3D reconstruction of articulated objects using motion consistency, an interpretable deformation model, and joint optimization, achieving state-of-the-art results with minimal supervision.

Contribution

It introduces a motion-based cycle loss for self-supervision, an interpretable mesh deformation model with local handles, and a joint optimization method for improved 3D reconstruction accuracy.

Findings

Achieves state-of-the-art 3D reconstructions for multiple categories.

Uses minimal video data for supervision, reducing data requirements.

Enhances reconstruction accuracy through joint mesh and camera optimization.

Abstract

Monocular 3D reconstruction of articulated object categories is challenging due to the lack of training data and the inherent ill-posedness of the problem. In this work we use video self-supervision, forcing the consistency of consecutive 3D reconstructions by a motion-based cycle loss. This largely improves both optimization-based and learning-based 3D mesh reconstruction. We further introduce an interpretable model of 3D template deformations that controls a 3D surface through the displacement of a small number of local, learnable handles. We formulate this operation as a structured layer relying on mesh-laplacian regularization and show that it can be trained in an end-to-end manner. We finally introduce a per-sample numerical optimisation approach that jointly optimises over mesh displacements and cameras within a video, boosting accuracy both for training and also as test time post-processing. While relying exclusively on a small set of videos collected per category for supervision, we obtain state-of-the-art reconstructions with diverse shapes, viewpoints and textures for multiple articulated object categories.