LSTM-Based Facial Performance Capture Using Embedding Between Expressions

TL;DR

This paper introduces an end-to-end facial performance capture framework that leverages a learned embedding space and LSTM to accurately model facial expressions and reduce jitter in monocular video tracking.

Contribution

The novel approach combines triplet loss-based embedding learning with LSTM for improved facial motion capture, especially for unseen expressions.

Findings

Better distinction of subtle lip movements

Significantly reduced jitter in mesh tracking

Effective generalization to unseen expressions

Abstract

We present a novel end-to-end framework for facial performance capture given a monocular video of an actor's face. Our framework are comprised of 2 parts. First, to extract the information in the frames, we optimize a triplet loss to learn the embedding space which ensures the semantically closer facial expressions are closer in the embedding space and the model can be transferred to distinguish the expressions that are not presented in the training dataset. Second, the embeddings are fed into an LSTM network to learn the deformation between frames. In the experiments, we demonstrated that compared to other methods, our method can distinguish the delicate motion around lips and significantly reduce jitters between the tracked meshes.