Learning to regulate 3D head shape by removing occluding hair from in-the-wild images

TL;DR

This paper introduces a novel method for 3D head shape reconstruction from in-the-wild images by removing occluding hair, enabling more accurate head shape modeling and landmark detection, with state-of-the-art results.

Contribution

It presents a new approach that removes occluding hair to improve 3D head shape reconstruction and introduces three novel training objectives for unsupervised learning.

Findings

Achieves state-of-the-art results on benchmark datasets.

Effectively models head shape despite occlusions.

Enables improved avatar creation and animation.

Abstract

Recent 3D face reconstruction methods reconstruct the entire head compared to earlier approaches which only model the face. Although these methods accurately reconstruct facial features, they do not explicitly regulate the upper part of the head. Extracting information about this part of the head is challenging due to varying degrees of occlusion by hair. We present a novel approach for modeling the upper head by removing occluding hair and reconstructing the skin, revealing information about the head shape. We introduce three objectives: 1) a dice consistency loss that enforces similarity between the overall head shape of the source and rendered image, 2) a scale consistency loss to ensure that head shape is accurately reproduced even if the upper part of the head is not visible, and 3) a 71 landmark detector trained using a moving average loss function to detect additional landmarks on the head. These objectives are used to train an encoder in an unsupervised manner to regress FLAME parameters from in-the-wild input images. Our unsupervised 3DMM model achieves state-of-the-art results on popular benchmarks and can be used to infer the head shape, facial features, and textures for direct use in animation or avatar creation.