PMVC: Data Augmentation-Based Prosody Modeling for Expressive Voice Conversion

TL;DR

This paper introduces PMVC, a novel voice conversion framework that effectively models and separates prosody, content, and timbre without text transcriptions, using a new speech augmentation method to enhance naturalness and similarity.

Contribution

The paper presents a new voice conversion model that disentangles prosody, content, and timbre without relying on transcriptions, utilizing a novel speech augmentation and mask-predict mechanism.

Findings

Improved naturalness of converted speech.

Enhanced similarity to target speaker.

Robust prosody extraction without transcriptions.

Abstract

Voice conversion as the style transfer task applied to speech, refers to converting one person's speech into a new speech that sounds like another person's. Up to now, there has been a lot of research devoted to better implementation of VC tasks. However, a good voice conversion model should not only match the timbre information of the target speaker, but also expressive information such as prosody, pace, pause, etc. In this context, prosody modeling is crucial for achieving expressive voice conversion that sounds natural and convincing. Unfortunately, prosody modeling is important but challenging, especially without text transcriptions. In this paper, we firstly propose a novel voice conversion framework named 'PMVC', which effectively separates and models the content, timbre, and prosodic information from the speech without text transcriptions. Specially, we introduce a new speech augmentation algorithm for robust prosody extraction. And building upon this, mask and predict mechanism is applied in the disentanglement of prosody and content information. The experimental results on the AIShell-3 corpus supports our improvement of naturalness and similarity of converted speech.