READ: Real-time and Efficient Asynchronous Diffusion for Audio-driven Talking Head Generation

TL;DR

READ introduces a real-time diffusion-transformer framework for audio-driven talking head generation, significantly improving speed while maintaining high quality and temporal consistency through innovative latent space modeling and asynchronous noise scheduling.

Contribution

The paper presents a novel real-time diffusion-transformer approach with a compressed latent space and asynchronous noise scheduler for efficient, high-quality talking head synthesis.

Findings

Outperforms state-of-the-art methods in speed and quality.

Achieves real-time inference with stable long-term video generation.

Maintains high audio-visual alignment and temporal consistency.

Abstract

The introduction of diffusion models has brought significant advances to the field of audio-driven talking head generation. However, the extremely slow inference speed severely limits the practical implementation of diffusion-based talking head generation models. In this study, we propose READ, a real-time diffusion-transformer-based talking head generation framework. Our approach first learns a spatiotemporal highly compressed video latent space via a temporal VAE, significantly reducing the token count to accelerate generation. To achieve better audio-visual alignment within this compressed latent space, a pre-trained Speech Autoencoder (SpeechAE) is proposed to generate temporally compressed speech latent codes corresponding to the video latent space. These latent representations are then modeled by a carefully designed Audio-to-Video Diffusion Transformer (A2V-DiT) backbone for efficient talking head synthesis. Furthermore, to ensure temporal consistency and accelerated inference in extended generation, we propose a novel asynchronous noise scheduler (ANS) for both the training and inference processes of our framework. The ANS leverages asynchronous add-noise and asynchronous motion-guided generation in the latent space, ensuring consistency in generated video clips. Experimental results demonstrate that READ outperforms state-of-the-art methods by generating competitive talking head videos with significantly reduced runtime, achieving an optimal balance between quality and speed while maintaining robust metric stability in long-time generation.