StarGAN-ZSVC: Towards Zero-Shot Voice Conversion in Low-Resource Contexts

TL;DR

This paper introduces StarGAN-ZSVC, a GAN-based model that enables real-time, zero-shot voice conversion without parallel data, trained on minimal data, and effective with unseen speakers.

Contribution

It extends StarGAN-VC by incorporating speaker embeddings for zero-shot conversion, demonstrating real-time performance with limited training data.

Findings

StarGAN-ZSVC outperforms AutoVC in zero-shot settings.

Real-time zero-shot voice conversion is achievable with minimal data.

Further scaling may improve high-resource zero-shot performance.

Abstract

Voice conversion is the task of converting a spoken utterance from a source speaker so that it appears to be said by a different target speaker while retaining the linguistic content of the utterance. Recent advances have led to major improvements in the quality of voice conversion systems. However, to be useful in a wider range of contexts, voice conversion systems would need to be (i) trainable without access to parallel data, (ii) work in a zero-shot setting where both the source and target speakers are unseen during training, and (iii) run in real time or faster. Recent techniques fulfil one or two of these requirements, but not all three. This paper extends recent voice conversion models based on generative adversarial networks (GANs), to satisfy all three of these conditions. We specifically extend the recent StarGAN-VC model by conditioning it on a speaker embedding (from a potentially unseen speaker). This allows the model to be used in a zero-shot setting, and we therefore call it StarGAN-ZSVC. We compare StarGAN-ZSVC against other voice conversion techniques in a low-resource setting using a small 9-minute training set. Compared to AutoVC -- another recent neural zero-shot approach -- we observe that StarGAN-ZSVC gives small improvements in the zero-shot setting, showing that real-time zero-shot voice conversion is possible even for a model trained on very little data. Further work is required to see whether scaling up StarGAN-ZSVC will also improve zero-shot voice conversion quality in high-resource contexts.