REST: Diffusion-based Real-time End-to-end Streaming Talking Head Generation via ID-Context Caching and Asynchronous Streaming Distillation

TL;DR

REST introduces a real-time, diffusion-based streaming talking head generation framework that combines novel caching and distillation strategies to achieve high efficiency and temporal coherence, enabling practical applications.

Contribution

The paper presents REST, a novel diffusion-based framework with ID-Context Caching and Asynchronous Streaming Distillation for real-time streaming talking head generation, addressing speed and coherence limitations.

Findings

REST achieves real-time performance surpassing state-of-the-art methods.

The ID-Context Cache maintains identity and temporal coherence during streaming.

Asynchronous Streaming Distillation improves temporal consistency and reduces error accumulation.

Abstract

Diffusion models have significantly advanced the field of talking head generation (THG). However, slow inference speeds and prevalent non-autoregressive paradigms severely constrain the application of diffusion-based THG models. In this study, we propose REST, a pioneering diffusion-based, real-time, end-to-end streaming audio-driven talking head generation framework. To support real-time end-to-end generation, a compact video latent space is first learned through a spatiotemporal variational autoencoder with a high compression ratio. Additionally, to enable semi-autoregressive streaming within the compact video latent space, we introduce an ID-Context Cache mechanism, which integrates ID-Sink and Context-Cache principles into key-value caching for maintaining identity consistency and temporal coherence during long-term streaming generation. Furthermore, an Asynchronous Streaming Distillation (ASD) strategy is proposed to mitigate error accumulation and enhance temporal consistency in streaming generation, leveraging a non-streaming teacher with an asynchronous noise schedule to supervise the streaming student. REST bridges the gap between autoregressive and diffusion-based approaches, achieving a breakthrough in efficiency for applications requiring real-time THG. Experimental results demonstrate that REST outperforms state-of-the-art methods in both generation speed and overall performance.