HDM-Net: Monocular Non-Rigid 3D Reconstruction with Learned Deformation Model

TL;DR

HDM-Net is a deep learning-based method for monocular non-rigid 3D reconstruction that learns deformation models from data, enabling accurate reconstruction without requiring templates or multiple frames.

Contribution

The paper introduces HDM-Net, a novel hybrid deep neural network that learns deformation models for monocular non-rigid 3D reconstruction, generalizing to unseen shapes, textures, and lighting conditions.

Findings

Demonstrates robustness to noisy data

Achieves accurate reconstruction without templates

Generalizes to unseen textures and illumination

Abstract

Monocular dense 3D reconstruction of deformable objects is a hard ill-posed problem in computer vision. Current techniques either require dense correspondences and rely on motion and deformation cues, or assume a highly accurate reconstruction (referred to as a template) of at least a single frame given in advance and operate in the manner of non-rigid tracking. Accurate computation of dense point tracks often requires multiple frames and might be computationally expensive. Availability of a template is a very strong prior which restricts system operation to a pre-defined environment and scenarios. In this work, we propose a new hybrid approach for monocular non-rigid reconstruction which we call Hybrid Deformation Model Network (HDM-Net). In our approach, deformation model is learned by a deep neural network, with a combination of domain-specific loss functions. We train the network with multiple states of a non-rigidly deforming structure with a known shape at rest. HDM-Net learns different reconstruction cues including texture-dependent surface deformations, shading and contours. We show generalisability of HDM-Net to states not presented in the training dataset, with unseen textures and under new illumination conditions. Experiments with noisy data and a comparison with other methods demonstrate robustness and accuracy of the proposed approach and suggest possible application scenarios of the new technique in interventional diagnostics and augmented reality.