Disentangling Timbre and Singing Style with Multi-singer Singing Synthesis System

TL;DR

This paper introduces a multi-singer singing synthesis system that independently models and controls singer identity, timbre, and singing style, enabling high-quality, expressive, and customizable singing voice generation.

Contribution

It extends single-singer models to multi-singer systems by designing a singer identity encoder and separate decoders for timbre and singing style, allowing independent control.

Findings

High-quality natural singing voice generation verified by user study.

Independent control of timbre and singing style demonstrated.

Enhanced expressiveness through style variation while fixing timbre.

Abstract

In this study, we define the identity of the singer with two independent concepts - timbre and singing style - and propose a multi-singer singing synthesis system that can model them separately. To this end, we extend our single-singer model into a multi-singer model in the following ways: first, we design a singer identity encoder that can adequately reflect the identity of a singer. Second, we use encoded singer identity to condition the two independent decoders that model timbre and singing style, respectively. Through a user study with the listening tests, we experimentally verify that the proposed framework is capable of generating a natural singing voice of high quality while independently controlling the timbre and singing style. Also, by using the method of changing singing styles while fixing the timbre, we suggest that our proposed network can produce a more expressive singing voice.