PR-RRN: Pairwise-Regularized Residual-Recursive Networks for Non-rigid Structure-from-Motion

TL;DR

PR-RRN introduces a neural network with residual-recursive structure and novel pairwise regularizations to improve non-rigid 3D shape and camera recovery from 2D keypoints, achieving state-of-the-art results.

Contribution

The paper presents a new neural network architecture with residual-recursive design and two innovative pairwise regularization losses for NRSfM.

Findings

Achieves state-of-the-art performance on CMU MOCAP and PASCAL3D+ datasets.

Effectively recovers 3D shape and camera parameters from 2D keypoints.

Introduces pairwise regularizations based on rigidity and consistency.

Abstract

We propose PR-RRN, a novel neural-network based method for Non-rigid Structure-from-Motion (NRSfM). PR-RRN consists of Residual-Recursive Networks (RRN) and two extra regularization losses. RRN is designed to effectively recover 3D shape and camera from 2D keypoints with novel residual-recursive structure. As NRSfM is a highly under-constrained problem, we propose two new pairwise regularization to further regularize the reconstruction. The Rigidity-based Pairwise Contrastive Loss regularizes the shape representation by encouraging higher similarity between the representations of high-rigidity pairs of frames than low-rigidity pairs. We propose minimum singular-value ratio to measure the pairwise rigidity. The Pairwise Consistency Loss enforces the reconstruction to be consistent when the estimated shapes and cameras are exchanged between pairs. Our approach achieves state-of-the-art performance on CMU MOCAP and PASCAL3D+ dataset.