SVP: Style-Enhanced Vivid Portrait Talking Head Diffusion Model

TL;DR

This paper introduces SVP, a diffusion model-based talking head generation framework that incorporates personalized style information, resulting in more diverse, vivid, and high-quality videos with controllable styles.

Contribution

The paper proposes a novel probabilistic style prior learning method and fine-tunes a pretrained Stable Diffusion model to enhance style diversity in talking head videos.

Findings

Generated videos are more diverse and vivid.

The method outperforms existing state-of-the-art approaches.

Flexible control over intrinsic styles is achieved.

Abstract

Talking Head Generation (THG), typically driven by audio, is an important and challenging task with broad application prospects in various fields such as digital humans, film production, and virtual reality. While diffusion model-based THG methods present high quality and stable content generation, they often overlook the intrinsic style which encompasses personalized features such as speaking habits and facial expressions of a video. As consequence, the generated video content lacks diversity and vividness, thus being limited in real life scenarios. To address these issues, we propose a novel framework named Style-Enhanced Vivid Portrait (SVP) which fully leverages style-related information in THG. Specifically, we first introduce the novel probabilistic style prior learning to model the intrinsic style as a Gaussian distribution using facial expressions and audio embedding. The distribution is learned through the 'bespoked' contrastive objective, effectively capturing the dynamic style information in each video. Then we finetune a pretrained Stable Diffusion (SD) model to inject the learned intrinsic style as a controlling signal via cross attention. Experiments show that our model generates diverse, vivid, and high-quality videos with flexible control over intrinsic styles, outperforming existing state-of-the-art methods.