Leveraging Diverse Semantic-based Audio Pretrained Models for Singing Voice Conversion

TL;DR

This paper explores the use of diverse semantic-based pretrained audio models for singing voice conversion, demonstrating that their complementary knowledge improves conversion quality, especially in real-world scenarios.

Contribution

It introduces a novel framework, DSFF-SVC, that fuses diverse semantic features from pretrained models to enhance singing voice conversion performance.

Findings

Diverse semantic models provide complementary information for SVC.

DSFF-SVC improves existing SVC models in real-world tasks.

The framework generalizes well across different SVC scenarios.

Abstract

Singing Voice Conversion (SVC) is a technique that enables any singer to perform any song. To achieve this, it is essential to obtain speaker-agnostic representations from the source audio, which poses a significant challenge. A common solution involves utilizing a semantic-based audio pretrained model as a feature extractor. However, the degree to which the extracted features can meet the SVC requirements remains an open question. This includes their capability to accurately model melody and lyrics, the speaker-independency of their underlying acoustic information, and their robustness for in-the-wild acoustic environments. In this study, we investigate the knowledge within classical semantic-based pretrained models in much detail. We discover that the knowledge of different models is diverse and can be complementary for SVC. Based on the above, we design a Singing Voice Conversion framework based on Diverse Semantic-based Feature Fusion (DSFF-SVC). Experimental results demonstrate that DSFF-SVC can be generalized and improve various existing SVC models, particularly in challenging real-world conversion tasks. Our demo website is available at https://diversesemanticsvc.github.io/.