Semi-supervised voice conversion with amortized variational inference

TL;DR

This paper presents a semi-supervised voice conversion method that leverages both parallel and non-parallel data, improving conversion quality especially when parallel data is scarce, by extending existing systems with semi-supervised training.

Contribution

It introduces a semi-supervised approach for voice conversion using amortized variational inference, enabling effective training with limited parallel data and utilizing non-parallel data to enhance performance.

Findings

Semi-supervised training outperforms fully supervised when parallel data is limited.

Increasing non-parallel data improves conversion quality.

Method extends existing voice conversion systems to semi-supervised settings.

Abstract

In this work we introduce a semi-supervised approach to the voice conversion problem, in which speech from a source speaker is converted into speech of a target speaker. The proposed method makes use of both parallel and non-parallel utterances from the source and target simultaneously during training. This approach can be used to extend existing parallel data voice conversion systems such that they can be trained with semi-supervision. We show that incorporating semi-supervision improves the voice conversion performance compared to fully supervised training when the number of parallel utterances is limited as in many practical applications. Additionally, we find that increasing the number non-parallel utterances used in training continues to improve performance when the amount of parallel training data is held constant.