Prosodic Alignment for off-screen automatic dubbing

TL;DR

This paper extends a prosodic alignment model to improve off-screen automatic dubbing, enhancing audiovisual coherence by better matching speech prosody in multilingual TED Talks and YouTube videos.

Contribution

The work introduces an extended prosodic alignment model that accommodates off-screen dubbing, reducing synchronization constraints and improving dubbing quality.

Findings

Significantly better subjective viewing experience with the extended model.

Effective across four language pairs and different video sources.

Improved audiovisual coherence in off-screen dubbing scenarios.

Abstract

The goal of automatic dubbing is to perform speech-to-speech translation while achieving audiovisual coherence. This entails isochrony, i.e., translating the original speech by also matching its prosodic structure into phrases and pauses, especially when the speaker's mouth is visible. In previous work, we introduced a prosodic alignment model to address isochrone or on-screen dubbing. In this work, we extend the prosodic alignment model to also address off-screen dubbing that requires less stringent synchronization constraints. We conduct experiments on four dubbing directions - English to French, Italian, German and Spanish - on a publicly available collection of TED Talks and on publicly available YouTube videos. Empirical results show that compared to our previous work the extended prosodic alignment model provides significantly better subjective viewing experience on videos in which on-screen and off-screen automatic dubbing is applied for sentences with speakers mouth visible and not visible, respectively.