VAST: Vivify Your Talking Avatar via Zero-Shot Expressive Facial Style Transfer

TL;DR

This paper introduces VAST, an unsupervised model that transfers expressive facial styles from arbitrary videos to neutral avatars, enhancing realism and expressiveness in talking face generation.

Contribution

The paper presents a novel zero-shot style transfer model with a style encoder, hybrid decoder, and style enhancer for more expressive talking avatars.

Findings

Improves avatar expressiveness and authenticity

Captures diverse facial styles from arbitrary videos

Enables zero-shot style transfer without supervision

Abstract

Current talking face generation methods mainly focus on speech-lip synchronization. However, insufficient investigation on the facial talking style leads to a lifeless and monotonous avatar. Most previous works fail to imitate expressive styles from arbitrary video prompts and ensure the authenticity of the generated video. This paper proposes an unsupervised variational style transfer model (VAST) to vivify the neutral photo-realistic avatars. Our model consists of three key components: a style encoder that extracts facial style representations from the given video prompts; a hybrid facial expression decoder to model accurate speech-related movements; a variational style enhancer that enhances the style space to be highly expressive and meaningful. With our essential designs on facial style learning, our model is able to flexibly capture the expressive facial style from arbitrary video prompts and transfer it onto a personalized image renderer in a zero-shot manner. Experimental results demonstrate the proposed approach contributes to a more vivid talking avatar with higher authenticity and richer expressiveness.