Warm Chat: Diffuse Emotion-aware Interactive Talking Head Avatar with Tree-Structured Guidance

TL;DR

Warm Chat introduces an emotion-aware talking head framework that uses large language models and a tree-structured dialogue guide to produce realistic, emotionally expressive avatars capable of bidirectional interaction.

Contribution

The paper presents a novel framework combining LLMs, a Transformer-based head mask generator, and a dialogue tree structure for emotion-adaptive, bidirectional talking head generation.

Findings

Produces temporally consistent, emotionally expressive avatars

Effectively models dialogue state transitions with a tree structure

Demonstrates superior performance in experiments

Abstract

Generative models have advanced rapidly, enabling impressive talking head generation that brings AI to life. However, most existing methods focus solely on one-way portrait animation. Even the few that support bidirectional conversational interactions lack precise emotion-adaptive capabilities, significantly limiting their practical applicability. In this paper, we propose Warm Chat, a novel emotion-aware talking head generation framework for dyadic interactions. Leveraging the dialogue generation capability of large language models (LLMs, e.g., GPT-4), our method produces temporally consistent virtual avatars with rich emotional variations that seamlessly transition between speaking and listening states. Specifically, we design a Transformer-based head mask generator that learns temporally consistent motion features in a latent mask space, capable of generating arbitrary-length, temporally consistent mask sequences to constrain head motions. Furthermore, we introduce an interactive talking tree structure to represent dialogue state transitions, where each tree node contains information such as child/parent/sibling nodes and the current character's emotional state. By performing reverse-level traversal, we extract rich historical emotional cues from the current node to guide expression synthesis. Extensive experiments demonstrate the superior performance and effectiveness of our method.