Learning in your voice: Non-parallel voice conversion based on speaker consistency loss

TL;DR

This paper introduces a non-parallel voice conversion method that leverages speaker consistency loss to improve voice conversion quality without needing parallel training data.

Contribution

It proposes a novel auto-encoder based model with disentanglement and speaker consistency loss, addressing training and conversion mismatch in non-parallel voice conversion.

Findings

Outperforms existing methods in subjective listening tests

Achieves higher objective speech quality metrics

Effectively preserves linguistic content during conversion

Abstract

In this paper, we propose a novel voice conversion strategy to resolve the mismatch between the training and conversion scenarios when parallel speech corpus is unavailable for training. Based on auto-encoder and disentanglement frameworks, we design the proposed model to extract identity and content representations while reconstructing the input speech signal itself. Since we use other speaker's identity information in the training process, the training philosophy is naturally matched with the objective of voice conversion process. In addition, we effectively design the disentanglement framework to reliably preserve linguistic information and to enhance the quality of converted speech signals. The superiority of the proposed method is shown in subjective listening tests as well as objective measures.