LCM-SVC: Latent Diffusion Model Based Singing Voice Conversion with Inference Acceleration via Latent Consistency Distillation

TL;DR

LCM-SVC introduces a latent diffusion model with latent consistency distillation to enable fast, high-quality singing voice conversion, significantly reducing inference time while preserving sound quality and timbre similarity.

Contribution

The paper presents a novel latent diffusion model with a distillation technique for efficient, high-quality singing voice conversion, achieving one-step inference.

Findings

Significantly reduced inference time compared to state-of-the-art models.

Maintained high sound quality and timbre similarity.

Effective one-step or few-step inference achieved.

Abstract

Any-to-any singing voice conversion (SVC) aims to transfer a target singer's timbre to other songs using a short voice sample. However many diffusion model based any-to-any SVC methods, which have achieved impressive results, usually suffered from low efficiency caused by a mass of inference steps. In this paper, we propose LCM-SVC, a latent consistency distillation (LCD) based latent diffusion model (LDM) to accelerate inference speed. We achieved one-step or few-step inference while maintaining the high performance by distilling a pre-trained LDM based SVC model, which had the advantages of timbre decoupling and sound quality. Experimental results show that our proposed method can significantly reduce the inference time and largely preserve the sound quality and timbre similarity comparing with other state-of-the-art SVC models. Audio samples are available at https://sounddemos.github.io/lcm-svc.