Towards Film-Making Production Dialogue, Narration, Monologue Adaptive Moving Dubbing Benchmarks

TL;DR

This paper introduces TA-Dubbing, a comprehensive benchmark for evaluating movie dubbing models across dialogue, narration, and actor adaptation, aiming to improve film production quality.

Contribution

It presents TA-Dubbing, a versatile, open-source benchmark that evaluates state-of-the-art dubbing models and multi-modal large language models in film production contexts.

Findings

TA-Dubbing covers multiple dimensions of movie dubbing evaluation.

It enables benchmarking of advanced dubbing models and large language models.

The benchmark is fully open-source and continuously updated.

Abstract

Movie dubbing has advanced significantly, yet assessing the real-world effectiveness of these models remains challenging. A comprehensive evaluation benchmark is crucial for two key reasons: 1) Existing metrics fail to fully capture the complexities of dialogue, narration, monologue, and actor adaptability in movie dubbing. 2) A practical evaluation system should offer valuable insights to improve movie dubbing quality and advancement in film production. To this end, we introduce Talking Adaptive Dubbing Benchmarks (TA-Dubbing), designed to improve film production by adapting to dialogue, narration, monologue, and actors in movie dubbing. TA-Dubbing offers several key advantages: 1) Comprehensive Dimensions: TA-Dubbing covers a variety of dimensions of movie dubbing, incorporating metric evaluations for both movie understanding and speech generation. 2) Versatile Benchmarking: TA-Dubbing is designed to evaluate state-of-the-art movie dubbing models and advanced multi-modal large language models. 3) Full Open-Sourcing: We fully open-source TA-Dubbing at https://github.com/woka- 0a/DeepDubber- V1 including all video suits, evaluation methods, annotations. We also continuously integrate new movie dubbing models into the TA-Dubbing leaderboard at https://github.com/woka- 0a/DeepDubber-V1 to drive forward the field of movie dubbing.