DiTalker: A Unified DiT-based Framework for High-Quality and Speaking Styles Controllable Portrait Animation

TL;DR

DiTalker is a novel unified framework based on diffusion transformers that enables high-quality, controllable portrait animation with dynamic speaking styles and precise lip synchronization.

Contribution

It introduces a new DiT-based architecture with separate style and emotion encoding modules and an audio-style fusion mechanism for improved controllability and quality.

Findings

Outperforms existing methods in lip synchronization accuracy.

Effectively controls speaking styles and head movements.

Preserves identity and background details in generated videos.

Abstract

Portrait animation aims to synthesize talking videos from a static reference face, conditioned on audio and style frame cues (e.g., emotion and head poses), while ensuring precise lip synchronization and faithful reproduction of speaking styles. Existing diffusion-based portrait animation methods primarily focus on lip synchronization or static emotion transformation, often overlooking dynamic styles such as head movements. Moreover, most of these methods rely on a dual U-Net architecture, which preserves identity consistency but incurs additional computational overhead. To this end, we propose DiTalker, a unified DiT-based framework for speaking style-controllable portrait animation. We design a Style-Emotion Encoding Module that employs two separate branches: a style branch extracting identity-specific style information (e.g., head poses and movements), and an emotion branch extracting identity-agnostic emotion features. We further introduce an Audio-Style Fusion Module that decouples audio and speaking styles via two parallel cross-attention layers, using these features to guide the animation process. To enhance the quality of results, we adopt and modify two optimization constraints: one to improve lip synchronization and the other to preserve fine-grained identity and background details. Extensive experiments demonstrate the superiority of DiTalker in terms of lip synchronization and speaking style controllability. Project Page: https://thenameishope.github.io/DiTalker/