SPEAK: Speech-Driven Pose and Emotion-Adjustable Talking Head Generation

TL;DR

SPEAK is a novel one-shot talking head generation framework that enables independent control of lip synchronization, facial emotions, and head poses, producing realistic and expressive facial animations from speech.

Contribution

The paper introduces IRFD for feature disentanglement and a face editing module for emotional and pose control, advancing one-shot talking head generation capabilities.

Findings

Ensures lip synchronization with speech.

Enables decoupled control of facial features.

Produces realistic, expressive facial animations.

Abstract

Most earlier researches on talking face generation have focused on the synchronization of lip motion and speech content. However, head pose and facial emotions are equally important characteristics of natural faces. While audio-driven talking face generation has seen notable advancements, existing methods either overlook facial emotions or are limited to specific individuals and cannot be applied to arbitrary subjects. In this paper, we propose a novel one-shot Talking Head Generation framework (SPEAK) that distinguishes itself from the general Talking Face Generation by enabling emotional and postural control. Specifically, we introduce Inter-Reconstructed Feature Disentanglement (IRFD) module to decouple facial features into three latent spaces. Then we design a face editing module that modifies speech content and facial latent codes into a single latent space. Subsequently, we present a novel generator that employs modified latent codes derived from the editing module to regulate emotional expression, head poses, and speech content in synthesizing facial animations. Extensive trials demonstrate that our method ensures lip synchronization with the audio while enabling decoupled control of facial features, it can generate realistic talking head with coordinated lip motions, authentic facial emotions, and smooth head movements. The demo video is available: https://anonymous.4open.science/r/SPEAK-8A22