DubWise: Video-Guided Speech Duration Control in Multimodal LLM-based Text-to-Speech for Dubbing

TL;DR

DubWise introduces a multimodal LLM-based TTS system that controls speech duration to ensure lip-sync accuracy in dubbing, even across different languages and texts, by leveraging cross-modal attention and duration control.

Contribution

The paper presents a novel multimodal LLM-based TTS method that aligns speech with lip movements across languages, improving lip sync and naturalness in dubbing applications.

Findings

Effective lip sync in cross-lingual dubbing scenarios.

Improved naturalness over state-of-the-art methods.

Successful application on Lip2Wav-Chemistry and LRS2 datasets.

Abstract

Audio-visual alignment after dubbing is a challenging research problem. To this end, we propose a novel method, DubWise Multi-modal Large Language Model (LLM)-based Text-to-Speech (TTS), which can control the speech duration of synthesized speech in such a way that it aligns well with the speakers lip movements given in the reference video even when the spoken text is different or in a different language. To accomplish this, we propose to utilize cross-modal attention techniques in a pre-trained GPT-based TTS. We combine linguistic tokens from text, speaker identity tokens via a voice cloning network, and video tokens via a proposed duration controller network. We demonstrate the effectiveness of our system on the Lip2Wav-Chemistry and LRS2 datasets. Also, the proposed method achieves improved lip sync and naturalness compared to the SOTAs for the same language but different text (i.e., non-parallel) and the different language, different text (i.e., cross-lingual) scenarios.