Imitating Arbitrary Talking Style for Realistic Audio-DrivenTalking Face Synthesis

TL;DR

This paper introduces an unsupervised framework for synthesizing realistic talking face videos that can imitate arbitrary talking styles from reference videos by learning style codes from 3DMM parameters.

Contribution

It proposes a novel style injection method that learns talking styles unsupervised from videos and can imitate and interpolate styles for more expressive talking face synthesis.

Findings

The framework can imitate styles without explicit annotations.

It produces more natural and expressive talking face videos.

Style interpolation enables new style generation.

Abstract

People talk with diversified styles. For one piece of speech, different talking styles exhibit significant differences in the facial and head pose movements. For example, the "excited" style usually talks with the mouth wide open, while the "solemn" style is more standardized and seldomly exhibits exaggerated motions. Due to such huge differences between different styles, it is necessary to incorporate the talking style into audio-driven talking face synthesis framework. In this paper, we propose to inject style into the talking face synthesis framework through imitating arbitrary talking style of the particular reference video. Specifically, we systematically investigate talking styles with our collected \textit{Ted-HD} dataset and construct style codes as several statistics of 3D morphable model~(3DMM) parameters. Afterwards, we devise a latent-style-fusion~(LSF) model to synthesize stylized talking faces by imitating talking styles from the style codes. We emphasize the following novel characteristics of our framework: (1) It doesn't require any annotation of the style, the talking style is learned in an unsupervised manner from talking videos in the wild. (2) It can imitate arbitrary styles from arbitrary videos, and the style codes can also be interpolated to generate new styles. Extensive experiments demonstrate that the proposed framework has the ability to synthesize more natural and expressive talking styles compared with baseline methods.