Learning Expressive Disentangled Speech Representations with Soft Speech Units and Adversarial Style Augmentation

TL;DR

This paper introduces SAVC, a novel expressive voice conversion framework that uses soft speech units and adversarial style augmentation to improve content preservation and prosody modeling, resulting in more natural and intelligible converted speech.

Contribution

The paper proposes a new voice conversion method utilizing soft speech units and adversarial style augmentation to better disentangle speaker, content, and prosody features.

Findings

Improved speech naturalness and intelligibility over previous methods.

Effective elimination of speaker information through adversarial style augmentation.

Implicit prosody modeling enhances expressive voice conversion.

Abstract

Voice conversion is the task to transform voice characteristics of source speech while preserving content information. Nowadays, self-supervised representation learning models are increasingly utilized in content extraction. However, in these representations, a lot of hidden speaker information leads to timbre leakage while the prosodic information of hidden units lacks use. To address these issues, we propose a novel framework for expressive voice conversion called "SAVC" based on soft speech units from HuBert-soft. Taking soft speech units as input, we design an attribute encoder to extract content and prosody features respectively. Specifically, we first introduce statistic perturbation imposed by adversarial style augmentation to eliminate speaker information. Then the prosody is implicitly modeled on soft speech units with knowledge distillation. Experiment results show that the intelligibility and naturalness of converted speech outperform previous work.