One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing

TL;DR

This paper introduces a neural talking-head synthesis model for video conferencing that uses a novel keypoint representation to generate realistic videos with low bandwidth, enabling head rotation and high visual quality.

Contribution

The paper presents a new keypoint-based neural synthesis method that outperforms existing approaches and enables efficient, high-quality video conferencing with head rotation capabilities.

Findings

Outperforms competing methods on benchmark datasets.

Achieves H.264 quality with only one-tenth bandwidth.

Enables head rotation during video synthesis.

Abstract

We propose a neural talking-head video synthesis model and demonstrate its application to video conferencing. Our model learns to synthesize a talking-head video using a source image containing the target person's appearance and a driving video that dictates the motion in the output. Our motion is encoded based on a novel keypoint representation, where the identity-specific and motion-related information is decomposed unsupervisedly. Extensive experimental validation shows that our model outperforms competing methods on benchmark datasets. Moreover, our compact keypoint representation enables a video conferencing system that achieves the same visual quality as the commercial H.264 standard while only using one-tenth of the bandwidth. Besides, we show our keypoint representation allows the user to rotate the head during synthesis, which is useful for simulating face-to-face video conferencing experiences.