MM-MovieDubber: Towards Multi-Modal Learning for Multi-Modal Movie Dubbing

TL;DR

This paper presents a multi-modal generative framework for movie dubbing that leverages visual and speech models to improve synchronization, style adaptation, and subtle detail handling, outperforming existing methods.

Contribution

Introduction of a multi-modal large vision-language model combined with speech generation for enhanced movie dubbing quality and versatility.

Findings

Improved metrics: LSE-D, SPK-SIM, EMO-SIM, MCD up to 19.08%.

Constructed a detailed movie dubbing dataset with annotations.

Outperforms state-of-the-art dubbing methods.

Abstract

Current movie dubbing technology can produce the desired speech using a reference voice and input video, maintaining perfect synchronization with the visuals while effectively conveying the intended emotions. However, crucial aspects of movie dubbing, including adaptation to various dubbing styles, effective handling of dialogue, narration, and monologues, as well as consideration of subtle details such as speaker age and gender, remain insufficiently explored. To tackle these challenges, we introduce a multi-modal generative framework. First, it utilizes a multi-modal large vision-language model (VLM) to analyze visual inputs, enabling the recognition of dubbing types and fine-grained attributes. Second, it produces high-quality dubbing using large speech generation models, guided by multi-modal inputs. Additionally, a movie dubbing dataset with annotations for dubbing types and subtle details is constructed to enhance movie understanding and improve dubbing quality for the proposed multi-modal framework. Experimental results across multiple benchmark datasets show superior performance compared to state-of-the-art (SOTA) methods. In details, the LSE-D, SPK-SIM, EMO-SIM, and MCD exhibit improvements of up to 1.09%, 8.80%, 19.08%, and 18.74%, respectively.