Conditional Deep Hierarchical Variational Autoencoder for Voice Conversion

TL;DR

This paper introduces a deep hierarchical VAE for voice conversion that enhances model expressiveness and speed, addressing redundancy in latent variables to improve naturalness and speaker similarity.

Contribution

It proposes a novel deep hierarchical VAE model with a non-autoregressive decoder and latent variable control using β-VAE, advancing voice conversion performance.

Findings

Achieved mean opinion scores above 3.5 for naturalness and similarity.

Outperformed existing autoencoder-based VC methods in experiments.

Enhanced model expressiveness improves voice conversion quality.

Abstract

Variational autoencoder-based voice conversion (VAE-VC) has the advantage of requiring only pairs of speeches and speaker labels for training. Unlike the majority of the research in VAE-VC which focuses on utilizing auxiliary losses or discretizing latent variables, this paper investigates how an increasing model expressiveness has benefits and impacts on the VAE-VC. Specifically, we first analyze VAE-VC from a rate-distortion perspective, and point out that model expressiveness is significant for VAE-VC because rate and distortion reflect similarity and naturalness of converted speeches. Based on the analysis, we propose a novel VC method using a deep hierarchical VAE, which has high model expressiveness as well as having fast conversion speed thanks to its non-autoregressive decoder. Also, our analysis reveals another problem that similarity can be degraded when the latent variable of VAEs has redundant information. We address the problem by controlling the information contained in the latent variable using $\beta$-VAE objective. In the experiment using VCTK corpus, the proposed method achieved mean opinion scores higher than 3.5 on both naturalness and similarity in inter-gender settings, which are higher than the scores of existing autoencoder-based VC methods.