Length Aware Speech Translation for Video Dubbing

TL;DR

This paper presents a length-aware speech translation model and decoding method designed for real-time video dubbing, improving synchronization between source and translated audio without sacrificing translation quality.

Contribution

It introduces a novel length-sensitive translation model and a length-aware beam search technique that efficiently produce translations of different lengths in a single pass.

Findings

Enhanced synchronization quality with MOS gains of 0.34 (Spanish) and 0.65 (Korean).

Maintained BLEU scores comparable to baseline models.

Enabled real-time, on-device video dubbing with length control.

Abstract

In video dubbing, aligning translated audio with the source audio is a significant challenge. Our focus is on achieving this efficiently, tailored for real-time, on-device video dubbing scenarios. We developed a phoneme-based end-to-end length-sensitive speech translation (LSST) model, which generates translations of varying lengths short, normal, and long using predefined tags. Additionally, we introduced length-aware beam search (LABS), an efficient approach to generate translations of different lengths in a single decoding pass. This approach maintained comparable BLEU scores compared to a baseline without length awareness while significantly enhancing synchronization quality between source and target audio, achieving a mean opinion score (MOS) gain of 0.34 for Spanish and 0.65 for Korean, respectively.