Noise-Robust Voice Conversion by Conditional Denoising Training Using Latent Variables of Recording Quality and Environment

TL;DR

This paper introduces a noise-robust voice conversion method that explicitly models recording quality and environment using latent variables, leading to improved speech naturalness in noisy conditions.

Contribution

The proposed training approach conditions voice conversion on latent variables for recording quality and environment, enhancing robustness to unseen noise conditions.

Findings

Improved speech quality in noisy environments.

Outperforms conventional denoising training methods.

Effective modeling of recording conditions enhances naturalness.

Abstract

We propose noise-robust voice conversion (VC) which takes into account the recording quality and environment of noisy source speech. Conventional denoising training improves the noise robustness of a VC model by learning noisy-to-clean VC process. However, the naturalness of the converted speech is limited when the noise of the source speech is unseen during the training. To this end, our proposed training conditions a VC model on two latent variables representing the recording quality and environment of the source speech. These latent variables are derived from deep neural networks pre-trained on recording quality assessment and acoustic scene classification and calculated in an utterance-wise or frame-wise manner. As a result, the trained VC model can explicitly learn information about speech degradation during the training. Objective and subjective evaluations show that our training improves the quality of the converted speech compared to the conventional training.