Enhancing the vocal range of single-speaker singing voice synthesis with melody-unsupervised pre-training

TL;DR

This paper introduces a melody-unsupervised pre-training approach using multi-singer data to expand the vocal range and improve the quality of single-speaker singing voice synthesis without requiring detailed annotations.

Contribution

It proposes a novel pre-training method that leverages multi-singer datasets with unsupervised pitch and phoneme information to enhance single-speaker SVS performance.

Findings

Outperforms baseline in sound quality and naturalness

Enhances vocal range without degrading timbre similarity

Introduces differentiable duration regulator and bi-directional flow model

Abstract

The single-speaker singing voice synthesis (SVS) usually underperforms at pitch values that are out of the singer's vocal range or associated with limited training samples. Based on our previous work, this work proposes a melody-unsupervised multi-speaker pre-training method conducted on a multi-singer dataset to enhance the vocal range of the single-speaker, while not degrading the timbre similarity. This pre-training method can be deployed to a large-scale multi-singer dataset, which only contains audio-and-lyrics pairs without phonemic timing information and pitch annotation. Specifically, in the pre-training step, we design a phoneme predictor to produce the frame-level phoneme probability vectors as the phonemic timing information and a speaker encoder to model the timbre variations of different singers, and directly estimate the frame-level f0 values from the audio to provide the pitch information. These pre-trained model parameters are delivered into the fine-tuning step as prior knowledge to enhance the single speaker's vocal range. Moreover, this work also contributes to improving the sound quality and rhythm naturalness of the synthesized singing voices. It is the first to introduce a differentiable duration regulator to improve the rhythm naturalness of the synthesized voice, and a bi-directional flow model to improve the sound quality. Experimental results verify that the proposed SVS system outperforms the baseline on both sound quality and naturalness.