Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head Synthesis

TL;DR

Ditto is a diffusion-based framework for talking head synthesis that offers fine-grained control and real-time inference, overcoming speed and control limitations of previous models.

Contribution

It introduces a novel diffusion transformer in motion space, optimized for disentanglement, control, and real-time streaming in talking head synthesis.

Findings

Generates high-quality talking head videos with vivid expressions.

Achieves real-time inference with low delay and streaming capability.

Demonstrates superior controllability compared to prior methods.

Abstract

Recent advances in diffusion models have endowed talking head synthesis with subtle expressions and vivid head movements, but have also led to slow inference speed and insufficient control over generated results. To address these issues, we propose Ditto, a diffusion-based talking head framework that enables fine-grained controls and real-time inference. Specifically, we utilize an off-the-shelf motion extractor and devise a diffusion transformer to generate representations in a specific motion space. We optimize the model architecture and training strategy to address the issues in generating motion representations, including insufficient disentanglement between motion and identity, and large internal discrepancies within the representation. Besides, we employ diverse conditional signals while establishing a mapping between motion representation and facial semantics, enabling control over the generation process and correction of the results. Moreover, we jointly optimize the holistic framework to enable streaming processing, real-time inference, and low first-frame delay, offering functionalities crucial for interactive applications such as AI assistants. Extensive experimental results demonstrate that Ditto generates compelling talking head videos and exhibits superiority in both controllability and real-time performance.