Towards Authentic Movie Dubbing with Retrieve-Augmented Director-Actor Interaction Learning

TL;DR

This paper introduces Authentic-Dubber, a novel approach to movie dubbing that simulates director-actor interactions using multimodal retrieval and progressive speech generation, leading to more emotionally authentic dubbing results.

Contribution

It proposes a new retrieve-augmented learning scheme with multimodal reference libraries, emotion-based retrieval, and progressive graph-based speech synthesis for authentic dubbing.

Findings

Improved emotional expressiveness in dubbing results.

Effective retrieval of relevant multimodal emotional cues.

Validated on V2C Animation benchmark with positive subjective and objective results.

Abstract

The automatic movie dubbing model generates vivid speech from given scripts, replicating a speaker's timbre from a brief timbre prompt while ensuring lip-sync with the silent video. Existing approaches simulate a simplified workflow where actors dub directly without preparation, overlooking the critical director-actor interaction. In contrast, authentic workflows involve a dynamic collaboration: directors actively engage with actors, guiding them to internalize the context cues, specifically emotion, before performance. To address this issue, we propose a new Retrieve-Augmented Director-Actor Interaction Learning scheme to achieve authentic movie dubbing, termed Authentic-Dubber, which contains three novel mechanisms: (1) We construct a multimodal Reference Footage library to simulate the learning footage provided by directors. Note that we integrate Large Language Models (LLMs) to achieve deep comprehension of emotional representations across multimodal signals. (2) To emulate how actors efficiently and comprehensively internalize director-provided footage during dubbing, we propose an Emotion-Similarity-based Retrieval-Augmentation strategy. This strategy retrieves the most relevant multimodal information that aligns with the target silent video. (3) We develop a Progressive Graph-based speech generation approach that incrementally incorporates the retrieved multimodal emotional knowledge, thereby simulating the actor's final dubbing process. The above mechanisms enable the Authentic-Dubber to faithfully replicate the authentic dubbing workflow, achieving comprehensive improvements in emotional expressiveness. Both subjective and objective evaluations on the V2C Animation benchmark dataset validate the effectiveness. The code and demos are available at https://github.com/AI-S2-Lab/Authentic-Dubber.