3D human tongue reconstruction from single "in-the-wild" images

TL;DR

This paper introduces the first end-to-end trainable system for 3D face and tongue reconstruction from single in-the-wild images, utilizing a novel GAN approach and a new diverse tongue dataset.

Contribution

The work presents a novel pipeline that jointly reconstructs 3D faces and tongues from single images, including a new GAN method and a large tongue dataset.

Findings

Accurately reconstructs 3D tongue and face in diverse conditions

Demonstrates robustness and realism in in-the-wild scenarios

Provides publicly available tongue dataset for future research

Abstract

3D face reconstruction from a single image is a task that has garnered increased interest in the Computer Vision community, especially due to its broad use in a number of applications such as realistic 3D avatar creation, pose invariant face recognition and face hallucination. Since the introduction of the 3D Morphable Model in the late 90's, we witnessed an explosion of research aiming at particularly tackling this task. Nevertheless, despite the increasing level of detail in the 3D face reconstructions from single images mainly attributed to deep learning advances, finer and highly deformable components of the face such as the tongue are still absent from all 3D face models in the literature, although being very important for the realness of the 3D avatar representations. In this work we present the first, to the best of our knowledge, end-to-end trainable pipeline that accurately reconstructs the 3D face together with the tongue. Moreover, we make this pipeline robust in "in-the-wild" images by introducing a novel GAN method tailored for 3D tongue surface generation. Finally, we make publicly available to the community the first diverse tongue dataset, consisting of 1,800 raw scans of 700 individuals varying in gender, age, and ethnicity backgrounds. As we demonstrate in an extensive series of quantitative as well as qualitative experiments, our model proves to be robust and realistically captures the 3D tongue structure, even in adverse "in-the-wild" conditions.