Noro: Noise-Robust One-shot Voice Conversion with Hidden Speaker Representation Learning

TL;DR

Noro is a noise-robust one-shot voice conversion system that maintains high performance in noisy real-world conditions by employing innovative encoding and contrastive learning techniques, and also advances speaker representation learning.

Contribution

The paper introduces Noro, a novel VC system with noise-robust components and demonstrates its effectiveness, along with exploring hidden speaker representations in VC.

Findings

Noro outperforms baseline in noisy and clean scenarios.

The reference encoder is competitive with self-supervised models.

Noro enhances real-world voice conversion applications.

Abstract

The effectiveness of one-shot voice conversion (VC) decreases in real-world scenarios where reference speeches, which are often sourced from the internet, contain various disturbances like background noise. To address this issue, we introduce Noro, a noise-robust one-shot VC system. Noro features innovative components tailored for VC using noisy reference speeches, including a dual-branch reference encoding module and a noise-agnostic contrastive speaker loss. Experimental results demonstrate that Noro outperforms our baseline system in both clean and noisy scenarios, highlighting its efficacy for real-world applications. Additionally, we investigate the hidden speaker representation capabilities of our baseline system by repurposing its reference encoder as a speaker encoder. The results show that it is competitive with several advanced self-supervised learning models for speaker representation under the SUPERB settings, highlighting the potential for advancing speaker representation learning through one-shot VC tasks.