Efficient conditioned face animation using frontally-viewed embedding

TL;DR

This paper introduces a multi-frames embedding method called Frontalizer for improved profile view rendering in face animation, achieving higher quality and lower landmark error, suitable for real-time low bandwidth applications.

Contribution

The paper presents Frontalizer, a novel multi-frames embedding that enhances profile view rendering and combines latent code conditioning, advancing face animation quality in low compute settings.

Findings

22% improvement in perceptual quality over baseline

73% reduction in landmark error over baseline

Real-time performance on iPhone 8 with low bandwidth

Abstract

As the quality of few shot facial animation from landmarks increases, new applications become possible, such as ultra low bandwidth video chat compression with a high degree of realism. However, there are some important challenges to tackle in order to improve the experience in real world conditions. In particular, the current approaches fail to represent profile views without distortions, while running in a low compute regime. We focus on this key problem by introducing a multi-frames embedding dubbed Frontalizer to improve profile views rendering. In addition to this core improvement, we explore the learning of a latent code conditioning generations along with landmarks to better convey facial expressions. Our dense models achieves 22% of improvement in perceptual quality and 73% reduction of landmark error over the first order model baseline on a subset of DFDC videos containing head movements. Declined with mobile architectures, our models outperform the previous state-of-the-art (improving perceptual quality by more than 16% and reducing landmark error by more than 47% on two datasets) while running on real time on iPhone 8 with very low bandwidth requirements.