Transformer VQ-VAE for Unsupervised Unit Discovery and Speech Synthesis: ZeroSpeech 2020 Challenge

TL;DR

This paper presents a Transformer-based VQ-VAE system for unsupervised speech unit discovery and synthesis, achieving zero-resource speech modeling without textual or phonetic labels, and participating in the ZeroSpeech 2020 challenge.

Contribution

We introduce a novel Transformer-based VQ-VAE and inverter for unsupervised speech unit discovery and synthesis, advancing zero-resource speech processing techniques.

Findings

Effective unsupervised subword unit extraction

High-quality speech synthesis from codebook units

Improved performance with regularization methods

Abstract

In this paper, we report our submitted system for the ZeroSpeech 2020 challenge on Track 2019. The main theme in this challenge is to build a speech synthesizer without any textual information or phonetic labels. In order to tackle those challenges, we build a system that must address two major components such as 1) given speech audio, extract subword units in an unsupervised way and 2) re-synthesize the audio from novel speakers. The system also needs to balance the codebook performance between the ABX error rate and the bitrate compression rate. Our main contribution here is we proposed Transformer-based VQ-VAE for unsupervised unit discovery and Transformer-based inverter for the speech synthesis given the extracted codebook. Additionally, we also explored several regularization methods to improve performance even further.