Deep Non-Rigid Structure from Motion with Missing Data

TL;DR

This paper introduces a deep learning-based hierarchical sparse coding model for Non-Rigid Structure from Motion that effectively handles missing data, shape variability, and large-scale problems without 3D supervision, outperforming existing methods.

Contribution

The paper presents a novel unsupervised deep neural network architecture for NRSfM that overcomes previous limitations in data scale and shape complexity, and introduces a new quality measure for reconstructability.

Findings

Outperforms state-of-the-art NRSfM methods in accuracy and robustness.

Handles missing and occluded 2D points without matrix completion.

Capable of reconstructing complex non-rigid shapes at large scale.

Abstract

Non-Rigid Structure from Motion (NRSfM) refers to the problem of reconstructing cameras and the 3D point cloud of a non-rigid object from an ensemble of images with 2D correspondences. Current NRSfM algorithms are limited from two perspectives: (i) the number of images, and (ii) the type of shape variability they can handle. These difficulties stem from the inherent conflict between the condition of the system and the degrees of freedom needing to be modeled -- which has hampered its practical utility for many applications within vision. In this paper we propose a novel hierarchical sparse coding model for NRSFM which can overcome (i) and (ii) to such an extent, that NRSFM can be applied to problems in vision previously thought too ill posed. Our approach is realized in practice as the training of an unsupervised deep neural network (DNN) auto-encoder with a unique architecture that is able to disentangle pose from 3D structure. Using modern deep learning computational platforms allows us to solve NRSfM problems at an unprecedented scale and shape complexity. Our approach has no 3D supervision, relying solely on 2D point correspondences. Further, our approach is also able to handle missing/occluded 2D points without the need for matrix completion. Extensive experiments demonstrate the impressive performance of our approach where we exhibit superior precision and robustness against all available state-of-the-art works in some instances by an order of magnitude. We further propose a new quality measure (based on the network weights) which circumvents the need for 3D ground-truth to ascertain the confidence we have in the reconstructability. We believe our work to be a significant advance over state-of-the-art in NRSFM.