Robust Training of Singing Voice Synthesis Using Prior and Posterior Uncertainty

TL;DR

This paper introduces an uncertainty-based training method for singing voice synthesis that enhances model robustness by using differentiable data augmentation and frame-level uncertainty prediction, especially in data-scarce scenarios.

Contribution

It presents a novel uncertainty-aware training framework combining prior and posterior uncertainty estimation for end-to-end singing voice synthesis models.

Findings

Improved synthesis quality on Opencpop and Ofuton-P datasets.

Enhanced performance in low-confidence and long-tail singing scenarios.

Effective handling of data scarcity and imbalanced pitch distributions.

Abstract

Singing voice synthesis (SVS) has seen remarkable advancements in recent years. However, compared to speech and general audio data, publicly available singing datasets remain limited. In practice, this data scarcity often leads to performance degradation in long-tail scenarios, such as imbalanced pitch distributions or rare singing styles. To mitigate these challenges, we propose uncertainty-based optimization to improve the training process of end-to-end SVS models. First, we introduce differentiable data augmentation in the adversarial training, which operates in a sample-wise manner to increase the prior uncertainty. Second, we incorporate a frame-level uncertainty prediction module that estimates the posterior uncertainty, enabling the model to allocate more learning capacity to low-confidence segments. Empirical results on the Opencpop and Ofuton-P, across Chinese and Japanese, demonstrate that our approach improves performance in various perspectives.