Robust Performance-driven 3D Face Tracking in Long Range Depth Scenes

TL;DR

This paper presents a robust hybrid 3D face tracking system that effectively tracks head pose and facial actions in RGBD videos, especially at large distances where point cloud quality is poor, by combining shape regression and joint optimization.

Contribution

It introduces a novel hybrid framework that combines a 3D shape regressor with joint 2D+3D optimization, enabling accurate long-range face tracking without pre-calibration.

Findings

Effective tracking at large scene depths.

Improved 3D face model reconstruction.

Enhanced depth map refinement for 3D tasks.

Abstract

We introduce a novel robust hybrid 3D face tracking framework from RGBD video streams, which is capable of tracking head pose and facial actions without pre-calibration or intervention from a user. In particular, we emphasize on improving the tracking performance in instances where the tracked subject is at a large distance from the cameras, and the quality of point cloud deteriorates severely. This is accomplished by the combination of a flexible 3D shape regressor and the joint 2D+3D optimization on shape parameters. Our approach fits facial blendshapes to the point cloud of the human head, while being driven by an efficient and rapid 3D shape regressor trained on generic RGB datasets. As an on-line tracking system, the identity of the unknown user is adapted on-the-fly resulting in improved 3D model reconstruction and consequently better tracking performance. The result is a robust RGBD face tracker, capable of handling a wide range of target scene depths, beyond those that can be afforded by traditional depth or RGB face trackers. Lastly, since the blendshape is not able to accurately recover the real facial shape, we use the tracked 3D face model as a prior in a novel filtering process to further refine the depth map for use in other tasks, such as 3D reconstruction.