Real-time Simultaneous 3D Head Modeling and Facial Motion Capture with an RGB-D camera

TL;DR

This paper introduces a real-time method for creating animated 3D head models and capturing facial motion using a consumer RGB-D camera, enabling instant reconstruction without prior training.

Contribution

It is the first to simultaneously provide comprehensive facial motion tracking and detailed 3D head modeling in real-time without pre-scanning or training.

Findings

Robust real-time facial motion capture across various expressions.

High-fidelity 3D head reconstructions for diverse subjects.

Efficient on-the-fly model refinement algorithm.

Abstract

We propose a method to build in real-time animated 3D head models using a consumer-grade RGB-D camera. Our proposed method is the first one to provide simultaneously comprehensive facial motion tracking and a detailed 3D model of the user's head. Anyone's head can be instantly reconstructed and his facial motion captured without requiring any training or pre-scanning. The user starts facing the camera with a neutral expression in the first frame, but is free to move, talk and change his face expression as he wills otherwise. The facial motion is captured using a blendshape animation model while geometric details are captured using a Deviation image mapped over the template mesh. We contribute with an efficient algorithm to grow and refine the deforming 3D model of the head on-the-fly and in real-time. We demonstrate robust and high-fidelity simultaneous facial motion capture and 3D head modeling results on a wide range of subjects with various head poses and facial expressions.