A Unit Enhancement and Guidance Framework for Audio-Driven Avatar Video Generation

TL;DR

This paper introduces PAHA, a novel framework for audio-driven avatar video generation that enhances regional guidance and consistency, significantly improving quality and efficiency over existing multi-stage methods.

Contribution

The paper proposes a new unit enhancement and guidance framework with two key methods, PAR and PCE, to improve visual quality and motion consistency in audio-driven avatar videos.

Findings

PAHA outperforms existing methods in audio-motion alignment.

The proposed classifiers improve regional consistency.

Experimental results validate the effectiveness of the framework.

Abstract

Audio-driven human animation technology is widely used in human-computer interaction, and the emergence of diffusion models has further advanced its development. Currently, most methods rely on multi-stage generation and intermediate representations, resulting in long inference time and issues with generation quality in specific foreground regions and audio-motion consistency. These shortcomings are primarily due to the lack of localized fine-grained supervised guidance. To address above challenges, we propose Parts-aware Audio-driven Human Animation, PAHA, a unit enhancement and guidance framework for audio-driven upper-body animation. We introduce two key methods: Parts-Aware Re-weighting (PAR) and Parts Consistency Enhancement (PCE). PAR dynamically adjusts regional training loss weights based on pose confidence scores, effectively improving visual quality. PCE constructs and trains diffusion-based regional audio-visual classifiers to improve the consistency of motion and co-speech audio. Afterwards, we design two novel inference guidance methods for the foregoing classifiers, Sequential Guidance (SG) and Differential Guidance (DG), to balance efficiency and quality respectively. Additionally, we build CNAS, the first public Chinese News Anchor Speech dataset, to advance research and validation in this field. Extensive experimental results and user studies demonstrate that PAHA significantly outperforms existing methods in audio-motion alignment and video-related evaluations. The codes and CNAS dataset will be released upon acceptance.