Fine-grained Video Dubbing Duration Alignment with Segment Supervised Preference Optimization

TL;DR

This paper introduces SSPO, a novel method for aligning speech durations in video dubbing by optimizing preferences at the segment level, improving synchronization between source and target speech.

Contribution

The paper presents the Segment Supervised Preference Optimization (SSPO) method, a new approach for fine-grained duration alignment in video dubbing using segment-wise sampling and loss functions.

Findings

SSPO outperforms existing methods in duration alignment accuracy.

The method effectively reduces audio-video synchronization issues.

Experimental results validate the superiority of SSPO in diverse video dubbing scenarios.

Abstract

Video dubbing aims to translate original speech in visual media programs from the source language to the target language, relying on neural machine translation and text-to-speech technologies. Due to varying information densities across languages, target speech often mismatches the source speech duration, causing audio-video synchronization issues that significantly impact viewer experience. In this study, we approach duration alignment in LLM-based video dubbing machine translation as a preference optimization problem. We propose the Segment Supervised Preference Optimization (SSPO) method, which employs a segment-wise sampling strategy and fine-grained loss to mitigate duration mismatches between source and target lines. Experimental results demonstrate that SSPO achieves superior performance in duration alignment tasks.