3D Face Tracking and Texture Fusion in the Wild

TL;DR

This paper introduces a real-time, fully automatic system for 3D face reconstruction from monocular videos in unconstrained environments, capturing expressions without person-specific training.

Contribution

It combines cascaded-regressor face tracking with 3D Morphable Model fitting and multi-frame texture fusion, enabling robust, real-time 3D face modeling in the wild.

Findings

Effective on 300-VW dataset

Real-time processing capability

Open source implementation available

Abstract

We present a fully automatic approach to real-time 3D face reconstruction from monocular in-the-wild videos. With the use of a cascaded-regressor based face tracking and a 3D Morphable Face Model shape fitting, we obtain a semi-dense 3D face shape. We further use the texture information from multiple frames to build a holistic 3D face representation from the video frames. Our system is able to capture facial expressions and does not require any person-specific training. We demonstrate the robustness of our approach on the challenging 300 Videos in the Wild (300-VW) dataset. Our real-time fitting framework is available as an open source library at http://4dface.org.