Non-rigid Structure-from-Motion: Temporally-smooth Procrustean Alignment and Spatially-variant Deformation Modeling

TL;DR

This paper introduces a novel spatial-temporal modeling approach for Non-rigid Structure-from-Motion that improves deformation modeling and camera motion estimation by using a temporally-smooth alignment and spatially adaptive low-rank constraints.

Contribution

It proposes a Temporally-smooth Procrustean Alignment module and a spatially-weighted low-rank constraint to better handle complex deformations and reduce motion ambiguity in NRSfM.

Findings

Outperforms existing low-rank based methods on various datasets.

Effectively models drastic spatially-variant deformations.

Remedies the need for complex reference shapes during alignment.

Abstract

Even though Non-rigid Structure-from-Motion (NRSfM) has been extensively studied and great progress has been made, there are still key challenges that hinder their broad real-world applications: 1) the inherent motion/rotation ambiguity requires either explicit camera motion recovery with extra constraint or complex Procrustean Alignment; 2) existing low-rank modeling of the global shape can over-penalize drastic deformations in the 3D shape sequence. This paper proposes to resolve the above issues from a spatial-temporal modeling perspective. First, we propose a novel Temporally-smooth Procrustean Alignment module that estimates 3D deforming shapes and adjusts the camera motion by aligning the 3D shape sequence consecutively. Our new alignment module remedies the requirement of complex reference 3D shape during alignment, which is more conductive to non-isotropic deformation modeling. Second, we propose a spatial-weighted approach to enforce the low-rank constraint adaptively at different locations to accommodate drastic spatially-variant deformation reconstruction better. Our modeling outperform existing low-rank based methods, and extensive experiments across different datasets validate the effectiveness of our method.