ToneUnit: A Speech Discretization Approach for Tonal Language Speech Synthesis

TL;DR

ToneUnit introduces a novel speech discretization framework for tonal languages like Mandarin, effectively addressing tone shift issues in synthesis and performing well even with limited annotated data.

Contribution

The paper proposes ToneUnit, a tone-aware speech discretization method using CTC supervision, improving tonal speech synthesis quality.

Findings

ToneUnit resolves tone shift in Mandarin speech synthesis.

Finite scalar quantization enhances ToneUnit's effectiveness.

Effective with minimal annotated data.

Abstract

Representing speech as discretized units has numerous benefits in supporting downstream spoken language processing tasks. However, the approach has been less explored in speech synthesis of tonal languages like Mandarin Chinese. Our preliminary experiments on Chinese speech synthesis reveal the issue of "tone shift", where a synthesized speech utterance contains correct base syllables but incorrect tones. To address the issue, we propose the ToneUnit framework, which leverages annotated data with tone labels as CTC supervision to learn tone-aware discrete speech units for Mandarin Chinese speech. Our findings indicate that the discrete units acquired through the TonUnit resolve the "tone shift" issue in synthesized Chinese speech and yield favorable results in English synthesis. Moreover, the experimental results suggest that finite scalar quantization enhances the effectiveness of ToneUnit. Notably, ToneUnit can work effectively even with minimal annotated data.