Exploring Disentanglement with Multilingual and Monolingual VQ-VAE

TL;DR

This paper investigates the use of disentangled phone and speaker representations from multilingual and monolingual VQ-VAE models for speech manipulation tasks like voice transformation and privacy masking.

Contribution

It introduces a novel approach to manipulate speech content and speaker identity using VQ-VAE representations, including a technique for content concealment.

Findings

VQ representations are effective for speech manipulation tasks.

Mixing speaker representations can create new voices.

Content masking preserves speaker identity and intelligibility.

Abstract

This work examines the content and usefulness of disentangled phone and speaker representations from two separately trained VQ-VAE systems: one trained on multilingual data and another trained on monolingual data. We explore the multi- and monolingual models using four small proof-of-concept tasks: copy-synthesis, voice transformation, linguistic code-switching, and content-based privacy masking. From these tasks, we reflect on how disentangled phone and speaker representations can be used to manipulate speech in a meaningful way. Our experiments demonstrate that the VQ representations are suitable for these tasks, including creating new voices by mixing speaker representations together. We also present our novel technique to conceal the content of targeted words within an utterance by manipulating phone VQ codes, while retaining speaker identity and intelligibility of surrounding words. Finally, we discuss recommendations for further increasing the viability of disentangled representations.