3D Face Hallucination from a Single Depth Frame

TL;DR

This paper introduces an automatic method to generate high-resolution 3D face models from a single depth image by matching and combining regional shapes from a large database, demonstrating robustness across various face types.

Contribution

The novel approach uses regional matching with a large 3D face database to produce detailed 3D face reconstructions from minimal depth data.

Findings

High-quality 3D face reconstructions from single depth frames.

Robustness across age, ethnicity, and facial expressions.

Outperforms existing shape estimation methods.

Abstract

We present an algorithm that takes a single frame of a person's face from a depth camera, e.g., Kinect, and produces a high-resolution 3D mesh of the input face. We leverage a dataset of 3D face meshes of 1204 distinct individuals ranging from age 3 to 40, captured in a neutral expression. We divide the input depth frame into semantically significant regions (eyes, nose, mouth, cheeks) and search the database for the best matching shape per region. We further combine the input depth frame with the matched database shapes into a single mesh that results in a high-resolution shape of the input person. Our system is fully automatic and uses only depth data for matching, making it invariant to imaging conditions. We evaluate our results using ground truth shapes, as well as compare to state-of-the-art shape estimation methods. We demonstrate the robustness of our local matching approach with high-quality reconstruction of faces that fall outside of the dataset span, e.g., faces older than 40 years old, facial expressions, and different ethnicities.