Disentangling the Prosody and Semantic Information with Pre-trained Model for In-Context Learning based Zero-Shot Voice Conversion

TL;DR

This paper introduces a zero-shot voice conversion system leveraging in-context learning and semantic tokens, enhanced with prosody embeddings to better preserve speech prosody, demonstrating improved speaker similarity and prosody retention.

Contribution

The paper proposes a novel ICL-based voice conversion system with a mask and reconstruction training strategy, integrating prosody embeddings to better preserve source speech prosody.

Findings

ICL-VC improves speaker similarity in voice conversion.

K-means is effective for tokenization across pre-trained models.

Incorporating prosody embeddings enhances prosody preservation.

Abstract

Voice conversion (VC) aims to modify the speaker's timbre while retaining speech content. Previous approaches have tokenized the outputs from self-supervised into semantic tokens, facilitating disentanglement of speech content information. Recently, in-context learning (ICL) has emerged in text-to-speech (TTS) systems for effectively modeling specific characteristics such as timbre through context conditioning. This paper proposes an ICL capability enhanced VC system (ICL-VC) employing a mask and reconstruction training strategy based on flow-matching generative models. Augmented with semantic tokens, our experiments on the LibriTTS dataset demonstrate that ICL-VC improves speaker similarity. Additionally, we find that k-means is a versatile tokenization method applicable to various pre-trained models. However, the ICL-VC system faces challenges in preserving the prosody of the source speech. To mitigate this issue, we propose incorporating prosody embeddings extracted from a pre-trained emotion recognition model into our system. Integration of prosody embeddings notably enhances the system's capability to preserve source speech prosody, as validated on the Emotional Speech Database.