EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model

TL;DR

This paper introduces EAMM, a novel model that generates one-shot emotional talking faces from audio and emotion videos, effectively capturing facial emotions and applicable to arbitrary subjects.

Contribution

The paper presents a new emotion-aware motion model with an implicit emotion displacement learner for realistic emotional talking face synthesis.

Findings

Successfully generates emotional talking faces with realistic expressions

Applicable to arbitrary subjects without subject-specific training

Outperforms existing methods in emotion and motion realism

Abstract

Although significant progress has been made to audio-driven talking face generation, existing methods either neglect facial emotion or cannot be applied to arbitrary subjects. In this paper, we propose the Emotion-Aware Motion Model (EAMM) to generate one-shot emotional talking faces by involving an emotion source video. Specifically, we first propose an Audio2Facial-Dynamics module, which renders talking faces from audio-driven unsupervised zero- and first-order key-points motion. Then through exploring the motion model's properties, we further propose an Implicit Emotion Displacement Learner to represent emotion-related facial dynamics as linearly additive displacements to the previously acquired motion representations. Comprehensive experiments demonstrate that by incorporating the results from both modules, our method can generate satisfactory talking face results on arbitrary subjects with realistic emotion patterns.