High quality voice conversion using prosodic and high-resolution spectral features

TL;DR

This paper presents a deep neural network framework that converts both spectral and prosodic features for high-quality voice conversion, utilizing autoencoder pretraining and segmental models to improve speech naturalness.

Contribution

The work introduces a novel DNN-based voice conversion method that jointly models high-resolution spectral and prosodic features with autoencoder pretraining and segmental prosody modeling.

Findings

Enhanced speech quality in objective evaluations

Improved naturalness in subjective listening tests

Effective modeling of spectral and prosodic features

Abstract

Voice conversion methods have advanced rapidly over the last decade. Studies have shown that speaker characteristics are captured by spectral feature as well as various prosodic features. Most existing conversion methods focus on the spectral feature as it directly represents the timbre characteristics, while some conversion methods have focused only on the prosodic feature represented by the fundamental frequency. In this paper, a comprehensive framework using deep neural networks to convert both timbre and prosodic features is proposed. The timbre feature is represented by a high-resolution spectral feature. The prosodic features include F0, intensity and duration. It is well known that DNN is useful as a tool to model high-dimensional features. In this work, we show that DNN initialized by our proposed autoencoder pretraining yields good quality DNN conversion models. This pretraining is tailor-made for voice conversion and leverages on autoencoder to capture the generic spectral shape of source speech. Additionally, our framework uses segmental DNN models to capture the evolution of the prosodic features over time. To reconstruct the converted speech, the spectral feature produced by the DNN model is combined with the three prosodic features produced by the DNN segmental models. Our experimental results show that the application of both prosodic and high-resolution spectral features leads to quality converted speech as measured by objective evaluation and subjective listening tests.