Direct Noisy Speech Modeling for Noisy-to-Noisy Voice Conversion

TL;DR

This paper introduces a noisy-to-noisy voice conversion framework that preserves background sounds while converting speaker identity, addressing distortion issues with an improved waveform modeling approach, and demonstrating significant performance gains.

Contribution

The paper proposes a novel noisy-to-noisy voice conversion framework with an improved module that directly models noisy speech, enhancing naturalness and similarity in background sound preservation.

Findings

Significant improvement over previous framework in naturalness scores

Achieves comparable speaker similarity to upper bound

Effectively preserves background sounds in noisy-to-noisy conversion

Abstract

Beyond the conventional voice conversion (VC) where the speaker information is converted without altering the linguistic content, the background sounds are informative and need to be retained in some real-world scenarios, such as VC in movie/video and VC in music where the voice is entangled with background sounds. As a new VC framework, we have developed a noisy-to-noisy (N2N) VC framework to convert the speaker's identity while preserving the background sounds. Although our framework consisting of a denoising module and a VC module well handles the background sounds, the VC module is sensitive to the distortion caused by the denoising module. To address this distortion issue, in this paper we propose the improved VC module to directly model the noisy speech waveform while controlling the background sounds. The experimental results have demonstrated that our improved framework significantly outperforms the previous one and achieves an acceptable score in terms of naturalness, while reaching comparable similarity performance to the upper bound of our framework.