Learning to Dub Movies via Hierarchical Prosody Models

TL;DR

This paper introduces a hierarchical prosody modeling approach for movie dubbing that leverages visual cues from lip, face, and scene to generate emotionally accurate speech matching video content.

Contribution

A novel hierarchical prosody model that integrates visual information for emotion-aware speech synthesis in movie dubbing tasks.

Findings

Outperforms previous methods on Chem and V2C benchmarks.

Effectively captures emotion and speech dynamics from video cues.

Generates speech with improved emotional and prosodic accuracy.

Abstract

Given a piece of text, a video clip and a reference audio, the movie dubbing (also known as visual voice clone V2C) task aims to generate speeches that match the speaker's emotion presented in the video using the desired speaker voice as reference. V2C is more challenging than conventional text-to-speech tasks as it additionally requires the generated speech to exactly match the varying emotions and speaking speed presented in the video. Unlike previous works, we propose a novel movie dubbing architecture to tackle these problems via hierarchical prosody modelling, which bridges the visual information to corresponding speech prosody from three aspects: lip, face, and scene. Specifically, we align lip movement to the speech duration, and convey facial expression to speech energy and pitch via attention mechanism based on valence and arousal representations inspired by recent psychology findings. Moreover, we design an emotion booster to capture the atmosphere from global video scenes. All these embeddings together are used to generate mel-spectrogram and then convert to speech waves via existing vocoder. Extensive experimental results on the Chem and V2C benchmark datasets demonstrate the favorable performance of the proposed method. The source code and trained models will be released to the public.