CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbing

TL;DR

CoSyncDiT introduces a novel diffusion transformer framework for movie dubbing that enhances lip-sync accuracy and naturalness by leveraging cognitive-inspired alignment and regularization techniques.

Contribution

It proposes a flow matching-based architecture with a joint regularization mechanism for improved synchronization and naturalness in movie dubbing.

Findings

Achieves state-of-the-art performance on standard and in-the-wild benchmarks.

Effectively maintains temporal and semantic consistency in speech synthesis.

Outperforms existing methods in lip-sync accuracy and naturalness.

Abstract

Movie dubbing aims to synthesize speech that preserves the vocal identity of a reference audio while synchronizing with the lip movements in a target video. Existing methods fail to achieve precise lip-sync and lack naturalness due to explicit alignment at the duration level. While implicit alignment solutions have emerged, they remain susceptible to interference from the reference audio, triggering timbre and pronunciation degradation in in-the-wild scenarios. In this paper, we propose a novel flow matching-based movie dubbing framework driven by the Cognitive Synchronous Diffusion Transformer (CoSync-DiT), inspired by the cognitive process of professional actors. This architecture progressively guides the noise-to-speech generative trajectory by executing acoustic style adapting, fine-grained visual calibrating, and time-aware context aligning. Furthermore, we design the Joint Semantic and Alignment Regularization (JSAR) mechanism to simultaneously constrain frame-level temporal consistency on the contextual outputs and semantic consistency on the flow hidden states, ensuring robust alignment. Extensive experiments on both standard benchmarks and challenging in-the-wild dubbing benchmarks demonstrate that our method achieves the state-of-the-art performance across multiple metrics.