Adversarial Multi-Task Learning for Disentangling Timbre and Pitch in Singing Voice Synthesis

TL;DR

This paper introduces a multi-task learning approach combining parametric and neural vocoder features to improve disentanglement of timbre and pitch in singing voice synthesis, resulting in more natural and controllable singing voices.

Contribution

It proposes a novel multi-task learning model that uses both parametric and mel-spectrogram features, enhancing voice quality and feature disentanglement in singing synthesis.

Findings

Generated singing voices are more natural than single-task models.

The model effectively disentangles timbre and pitch components.

It outperforms conventional parametric vocoder-based models.

Abstract

Recently, deep learning-based generative models have been introduced to generate singing voices. One approach is to predict the parametric vocoder features consisting of explicit speech parameters. This approach has the advantage that the meaning of each feature is explicitly distinguished. Another approach is to predict mel-spectrograms for a neural vocoder. However, parametric vocoders have limitations of voice quality and the mel-spectrogram features are difficult to model because the timbre and pitch information are entangled. In this study, we propose a singing voice synthesis model with multi-task learning to use both approaches -- acoustic features for a parametric vocoder and mel-spectrograms for a neural vocoder. By using the parametric vocoder features as auxiliary features, the proposed model can efficiently disentangle and control the timbre and pitch components of the mel-spectrogram. Moreover, a generative adversarial network framework is applied to improve the quality of singing voices in a multi-singer model. Experimental results demonstrate that our proposed model can generate more natural singing voices than the single-task models, while performing better than the conventional parametric vocoder-based model.