FLAP: Fully-controllable Audio-driven Portrait Video Generation through 3D head conditioned diffusion model

TL;DR

FLAP is a novel diffusion-based method that enables fully controllable, realistic audio-driven portrait video generation with independent control over head pose and facial expressions, suitable for practical applications.

Contribution

Introducing FLAP, which integrates explicit 3D parameters into diffusion models for end-to-end controllable portrait video synthesis from audio.

Findings

Outperforms recent models in naturalness and controllability

Allows independent manipulation of head pose and facial expressions

Demonstrates flexibility with existing 3D head generation methods

Abstract

Diffusion-based video generation techniques have significantly improved zero-shot talking-head avatar generation, enhancing the naturalness of both head motion and facial expressions. However, existing methods suffer from poor controllability, making them less applicable to real-world scenarios such as filmmaking and live streaming for e-commerce. To address this limitation, we propose FLAP, a novel approach that integrates explicit 3D intermediate parameters (head poses and facial expressions) into the diffusion model for end-to-end generation of realistic portrait videos. The proposed architecture allows the model to generate vivid portrait videos from audio while simultaneously incorporating additional control signals, such as head rotation angles and eye-blinking frequency. Furthermore, the decoupling of head pose and facial expression allows for independent control of each, offering precise manipulation of both the avatar's pose and facial expressions. We also demonstrate its flexibility in integrating with existing 3D head generation methods, bridging the gap between 3D model-based approaches and end-to-end diffusion techniques. Extensive experiments show that our method outperforms recent audio-driven portrait video models in both naturalness and controllability.