LDM-SVC: Latent Diffusion Model Based Zero-Shot Any-to-Any Singing Voice Conversion with Singer Guidance

TL;DR

This paper introduces LDM-SVC, a novel zero-shot singing voice conversion method using latent diffusion models and singer guidance to effectively reduce timbre leakage and improve voice conversion quality.

Contribution

It proposes a latent diffusion model for singing voice conversion and a singer guidance training method to suppress original singer's timbre, advancing zero-shot SVC techniques.

Findings

Outperforms previous methods in timbre similarity evaluations.

Effectively reduces timbre leakage during voice conversion.

Demonstrates superior subjective and objective results.

Abstract

Any-to-any singing voice conversion (SVC) is an interesting audio editing technique, aiming to convert the singing voice of one singer into that of another, given only a few seconds of singing data. However, during the conversion process, the issue of timbre leakage is inevitable: the converted singing voice still sounds like the original singer's voice. To tackle this, we propose a latent diffusion model for SVC (LDM-SVC) in this work, which attempts to perform SVC in the latent space using an LDM. We pretrain a variational autoencoder structure using the noted open-source So-VITS-SVC project based on the VITS framework, which is then used for the LDM training. Besides, we propose a singer guidance training method based on classifier-free guidance to further suppress the timbre of the original singer. Experimental results show the superiority of the proposed method over previous works in both subjective and objective evaluations of timbre similarity.