Creating Speech-to-Speech Corpus from Dubbed Series

TL;DR

This paper presents an unsupervised method to create speech-to-speech corpora from dubbed series by aligning segments across languages using multimedia processing techniques, enabling resource creation for speech translation research.

Contribution

The authors introduce a novel unsupervised pipeline that leverages video, speech recognition, and translation to generate aligned speech corpora from dubbed content, applicable across languages.

Findings

Generated 17 hours of paired segments from 36 hours of dubbed series.

Achieved approximately 70% accuracy in segment alignment based on human evaluation.

Method is robust across different language pairs, demonstrated on TR-AR and EN-AR datasets.

Abstract

Dubbed series are gaining a lot of popularity in recent years with strong support from major media service providers. Such popularity is fueled by studies that showed that dubbed versions of TV shows are more popular than their subtitled equivalents. We propose an unsupervised approach to construct speech-to-speech corpus, aligned on short segment levels, to produce a parallel speech corpus in the source- and target- languages. Our methodology exploits video frames, speech recognition, machine translation, and noisy frames removal algorithms to match segments in both languages. To verify the performance of the proposed method, we apply it on long and short dubbed clips. Out of 36 hours TR-AR dubbed series, our pipeline was able to generate 17 hours of paired segments, which is about 47% of the corpus. We applied our method on another language pair, EN-AR, to ensure it is robust enough and not tuned for a specific language or a specific corpus. Regardless of the language pairs, the accuracy of the paired segments was around 70% when evaluated using human subjective evaluation. The corpus will be freely available for the research community.