RAP: Real-time Audio-driven Portrait Animation with Video Diffusion Transformer

TL;DR

RAP is a novel framework that enables real-time, high-quality audio-driven portrait animation by combining hybrid attention and a static-dynamic paradigm, overcoming computational and detail-preservation challenges.

Contribution

It introduces a hybrid attention mechanism and a static-dynamic training-inference paradigm for real-time, detailed, and synchronized portrait animation without explicit motion supervision.

Findings

Achieves state-of-the-art performance in real-time portrait animation.

Maintains high visual fidelity and precise audio-visual synchronization.

Operates efficiently under strict latency and memory constraints.

Abstract

Audio-driven portrait animation aims to synthesize realistic and natural talking head videos from an input audio signal and a single reference image. While existing methods achieve high-quality results by leveraging high-dimensional intermediate representations and explicitly modeling motion dynamics, their computational complexity renders them unsuitable for real-time deployment. Real-time inference imposes stringent latency and memory constraints, often necessitating the use of highly compressed latent representations. However, operating in such compact spaces hinders the preservation of fine-grained spatiotemporal details, thereby complicating audio-visual synchronization RAP (Real-time Audio-driven Portrait animation), a unified framework for generating high-quality talking portraits under real-time constraints. Specifically, RAP introduces a hybrid attention mechanism for fine-grained audio control, and a static-dynamic training-inference paradigm that avoids explicit motion supervision. Through these techniques, RAP achieves precise audio-driven control, mitigates long-term temporal drift, and maintains high visual fidelity. Extensive experiments demonstrate that RAP achieves state-of-the-art performance while operating under real-time constraints.