3D Face Pose and Animation Tracking via Eigen-Decomposition based Bayesian Approach

TL;DR

This paper introduces a Bayesian method utilizing eigen-decomposition for accurate 3D face pose and animation tracking from monocular video, leveraging a synthetic database and SIFT features for improved landmark localization.

Contribution

The paper proposes a novel Bayesian approach with eigen-decomposition for simultaneous face pose and animation tracking using monocular video and synthetic training data.

Findings

Outperforms six existing algorithms in landmark localization accuracy

Provides a good balance between pose estimation and animation tracking

Shows promising results on BUFT and Talking Face datasets

Abstract

This paper presents a new method to track both the face pose and the face animation with a monocular camera. The approach is based on the 3D face model CANDIDE and on the SIFT (Scale Invariant Feature Transform) descriptors, extracted around a few given landmarks (26 selected vertices of CANDIDE model) with a Bayesian approach. The training phase is performed on a synthetic database generated from the first video frame. At each current frame, the face pose and animation parameters are estimated via a Bayesian approach, with a Gaussian prior and a Gaussian likelihood function whose the mean and the covariance matrix eigenvalues are updated from the previous frame using eigen decomposition. Numerical results on pose estimation and landmark locations are reported using the Boston University Face Tracking (BUFT) database and Talking Face video. They show that our approach, compared to six other published algorithms, provides a very good compromise and presents a promising perspective due to the good results in terms of landmark localization.