# HDM-Net: Monocular Non-Rigid 3D Reconstruction with Learned Deformation Model

## TL;DR

HDM-Net is a deep learning-based method for monocular non-rigid 3D reconstruction that learns deformation models from data, enabling accurate reconstruction without requiring templates or multiple frames.

## Contribution

The paper introduces HDM-Net, a novel hybrid deep neural network that learns deformation models for monocular non-rigid 3D reconstruction, generalizing to unseen shapes, textures, and lighting conditions.

## Key findings

- Demonstrates robustness to noisy data
- Achieves accurate reconstruction without templates
- Generalizes to unseen textures and illumination

## Abstract

Monocular dense 3D reconstruction of deformable objects is a hard ill-posed problem in computer vision. Current techniques either require dense correspondences and rely on motion and deformation cues, or assume a highly accurate reconstruction (referred to as a template) of at least a single frame given in advance and operate in the manner of non-rigid tracking. Accurate computation of dense point tracks often requires multiple frames and might be computationally expensive. Availability of a template is a very strong prior which restricts system operation to a pre-defined environment and scenarios. In this work, we propose a new hybrid approach for monocular non-rigid reconstruction which we call Hybrid Deformation Model Network (HDM-Net). In our approach, deformation model is learned by a deep neural network, with a combination of domain-specific loss functions. We train the network with multiple states of a non-rigidly deforming structure with a known shape at rest. HDM-Net learns different reconstruction cues including texture-dependent surface deformations, shading and contours. We show generalisability of HDM-Net to states not presented in the training dataset, with unseen textures and under new illumination conditions. Experiments with noisy data and a comparison with other methods demonstrate robustness and accuracy of the proposed approach and suggest possible application scenarios of the new technique in interventional diagnostics and augmented reality.