Text-free non-parallel many-to-many voice conversion using normalising flows

TL;DR

This paper explores the use of normalising flows for non-parallel, text-free voice conversion, demonstrating lossless speech encoding and improved performance over existing methods, with insights on prior training strategies.

Contribution

It introduces a novel application of normalising flows for voice conversion, comparing text-conditioned and text-free scenarios, and analyzing the impact of prior training methods.

Findings

Normalising flows enable lossless speech encoding for VC.

No performance degradation between text-free and text-conditioned VC.

Joint training of priors negatively affects text-free VC quality.

Abstract

Non-parallel voice conversion (VC) is typically achieved using lossy representations of the source speech. However, ensuring only speaker identity information is dropped whilst all other information from the source speech is retained is a large challenge. This is particularly challenging in the scenario where at inference-time we have no knowledge of the text being read, i.e., text-free VC. To mitigate this, we investigate information-preserving VC approaches. Normalising flows have gained attention for text-to-speech synthesis, however have been under-explored for VC. Flows utilize invertible functions to learn the likelihood of the data, thus provide a lossless encoding of speech. We investigate normalising flows for VC in both text-conditioned and text-free scenarios. Furthermore, for text-free VC we compare pre-trained and jointly-learnt priors. Flow-based VC evaluations show no degradation between text-free and text-conditioned VC, resulting in improvements over the state-of-the-art. Also, joint-training of the prior is found to negatively impact text-free VC quality.