End-to-End Voice Conversion with Information Perturbation

TL;DR

This paper introduces an end-to-end voice conversion method that uses information perturbation and specialized encoders to improve naturalness, speaker similarity, and prosody transfer in converted speech.

Contribution

It proposes a novel end-to-end framework with information perturbation and a speaker-related pitch encoder for high-quality voice conversion.

Findings

Outperforms state-of-the-art models in naturalness and speaker similarity

Effectively transfers source prosody and maintains target speaker timbre

Enhances speech intelligibility and quality

Abstract

The ideal goal of voice conversion is to convert the source speaker's speech to sound naturally like the target speaker while maintaining the linguistic content and the prosody of the source speech. However, current approaches are insufficient to achieve comprehensive source prosody transfer and target speaker timbre preservation in the converted speech, and the quality of the converted speech is also unsatisfied due to the mismatch between the acoustic model and the vocoder. In this paper, we leverage the recent advances in information perturbation and propose a fully end-to-end approach to conduct high-quality voice conversion. We first adopt information perturbation to remove speaker-related information in the source speech to disentangle speaker timbre and linguistic content and thus the linguistic information is subsequently modeled by a content encoder. To better transfer the prosody of the source speech to the target, we particularly introduce a speaker-related pitch encoder which can maintain the general pitch pattern of the source speaker while flexibly modifying the pitch intensity of the generated speech. Finally, one-shot voice conversion is set up through continuous speaker space modeling. Experimental results indicate that the proposed end-to-end approach significantly outperforms the state-of-the-art models in terms of intelligibility, naturalness, and speaker similarity.