Hybrid Approach for 3D Head Reconstruction: Using Neural Networks and Visual Geometry

TL;DR

This paper introduces a hybrid deep learning and geometric method for 3D head reconstruction from images, achieving state-of-the-art results even when trained solely on synthetic data.

Contribution

It presents a novel encoder-decoder network that predicts normals and landmarks for 3D head reconstruction, combining deep learning with geometric optimization.

Findings

Achieves high accuracy in 3D head reconstruction from single and multiple images.

Successfully generalizes from synthetic training data to real-world images.

Outperforms existing methods in qualitative and quantitative evaluations.

Abstract

Recovering the 3D geometric structure of a face from a single input image is a challenging active research area in computer vision. In this paper, we present a novel method for reconstructing 3D heads from a single or multiple image(s) using a hybrid approach based on deep learning and geometric techniques. We propose an encoder-decoder network based on the U-net architecture and trained on synthetic data only. It predicts both pixel-wise normal vectors and landmarks maps from a single input photo. Landmarks are used for the pose computation and the initialization of the optimization problem, which, in turn, reconstructs the 3D head geometry by using a parametric morphable model and normal vector fields. State-of-the-art results are achieved through qualitative and quantitative evaluation tests on both single and multi-view settings. Despite the fact that the model was trained only on synthetic data, it successfully recovers 3D geometry and precise poses for real-world images.