Learning Frame-Wise Emotion Intensity for Audio-Driven Talking-Head Generation

TL;DR

This paper introduces a novel framework for generating emotionally expressive talking-head videos by modeling continuous emotion intensity fluctuations, capturing subtle dynamic changes during speech for more realistic and expressive outputs.

Contribution

It proposes a continuous emotion latent space and an audio-to-intensity predictor trained with pseudo-labels, enabling precise control and realistic modeling of emotion intensity dynamics.

Findings

Effective in capturing emotion intensity fluctuations

Enhances realism and expressiveness of generated talking-heads

Validated through extensive experiments

Abstract

Human emotional expression is inherently dynamic, complex, and fluid, characterized by smooth transitions in intensity throughout verbal communication. However, the modeling of such intensity fluctuations has been largely overlooked by previous audio-driven talking-head generation methods, which often results in static emotional outputs. In this paper, we explore how emotion intensity fluctuates during speech, proposing a method for capturing and generating these subtle shifts for talking-head generation. Specifically, we develop a talking-head framework that is capable of generating a variety of emotions with precise control over intensity levels. This is achieved by learning a continuous emotion latent space, where emotion types are encoded within latent orientations and emotion intensity is reflected in latent norms. In addition, to capture the dynamic intensity fluctuations, we adopt an audio-to-intensity predictor by considering the speaking tone that reflects the intensity. The training signals for this predictor are obtained through our emotion-agnostic intensity pseudo-labeling method without the need of frame-wise intensity labeling. Extensive experiments and analyses validate the effectiveness of our proposed method in accurately capturing and reproducing emotion intensity fluctuations in talking-head generation, thereby significantly enhancing the expressiveness and realism of the generated outputs.