Pureformer-VC: Non-parallel One-Shot Voice Conversion with Pure Transformer Blocks and Triplet Discriminative Training

TL;DR

Pureformer-VC introduces a novel non-parallel, one-shot voice conversion framework utilizing pure Transformer blocks and triplet discriminative training, achieving improved objective metrics and comparable subjective quality.

Contribution

The paper proposes a new voice conversion model with pure Transformer-based architecture and triplet loss, enhancing disentanglement and style transfer without parallel data.

Findings

Achieves comparable subjective voice quality to existing methods.

Shows improvements in objective speech conversion metrics.

Effective disentanglement of speaker characteristics.

Abstract

One-shot voice conversion(VC) aims to change the timbre of any source speech to match that of the target speaker with only one speech sample. Existing style transfer-based VC methods relied on speech representation disentanglement and suffered from accurately and independently encoding each speech component and recomposing back to converted speech effectively. To tackle this, we proposed Pureformer-VC, which utilizes Conformer blocks to build a disentangled encoder, and Zipformer blocks to build a style transfer decoder as the generator. In the decoder, we used effective styleformer blocks to integrate speaker characteristics effectively into the generated speech. The models used the generative VAE loss for encoding components and triplet loss for unsupervised discriminative training. We applied the styleformer method to Zipformer's shared weights for style transfer. The experimental results show that the proposed model achieves comparable subjective scores and exhibits improvements in objective metrics compared to existing methods in a one-shot voice conversion scenario.