EMO2: End-Effector Guided Audio-Driven Avatar Video Generation

TL;DR

This paper introduces EMO2, a two-stage audio-driven avatar video generation method that produces expressive facial expressions and hand gestures, outperforming existing approaches in quality and synchronization.

Contribution

The paper presents a novel two-stage framework for co-speech gesture generation, focusing on hand pose synthesis from audio and integrating it with video synthesis using diffusion models.

Findings

Outperforms state-of-the-art methods like CyberHost and Vlogger.

Achieves higher visual quality and synchronization accuracy.

Effectively generates expressive facial and hand gestures from audio.

Abstract

In this paper, we propose a novel audio-driven talking head method capable of simultaneously generating highly expressive facial expressions and hand gestures. Unlike existing methods that focus on generating full-body or half-body poses, we investigate the challenges of co-speech gesture generation and identify the weak correspondence between audio features and full-body gestures as a key limitation. To address this, we redefine the task as a two-stage process. In the first stage, we generate hand poses directly from audio input, leveraging the strong correlation between audio signals and hand movements. In the second stage, we employ a diffusion model to synthesize video frames, incorporating the hand poses generated in the first stage to produce realistic facial expressions and body movements. Our experimental results demonstrate that the proposed method outperforms state-of-the-art approaches, such as CyberHost and Vlogger, in terms of both visual quality and synchronization accuracy. This work provides a new perspective on audio-driven gesture generation and a robust framework for creating expressive and natural talking head animations.