SLMGAN: Exploiting Speech Language Model Representations for Unsupervised Zero-Shot Voice Conversion in GANs

TL;DR

SLMGAN introduces a novel zero-shot voice conversion method leveraging speech language model representations within a GAN framework, achieving superior naturalness and comparable similarity without requiring text labels during training.

Contribution

The paper presents SLMGAN, a new approach that incorporates SLM-based discriminators and a feature matching loss for unsupervised zero-shot voice conversion, extending prior GAN models.

Findings

Outperforms state-of-the-art zero-shot voice conversion models in naturalness.

Achieves comparable speaker similarity to existing methods.

Operates without text labels during training.

Abstract

In recent years, large-scale pre-trained speech language models (SLMs) have demonstrated remarkable advancements in various generative speech modeling applications, such as text-to-speech synthesis, voice conversion, and speech enhancement. These applications typically involve mapping text or speech inputs to pre-trained SLM representations, from which target speech is decoded. This paper introduces a new approach, SLMGAN, to leverage SLM representations for discriminative tasks within the generative adversarial network (GAN) framework, specifically for voice conversion. Building upon StarGANv2-VC, we add our novel SLM-based WavLM discriminators on top of the mel-based discriminators along with our newly designed SLM feature matching loss function, resulting in an unsupervised zero-shot voice conversion system that does not require text labels during training. Subjective evaluation results show that SLMGAN outperforms existing state-of-the-art zero-shot voice conversion models in terms of naturalness and achieves comparable similarity, highlighting the potential of SLM-based discriminators for related applications.