Probing the phonetic and phonological knowledge of tones in Mandarin TTS models

TL;DR

This paper investigates how well Mandarin TTS models understand and reproduce tonal phonetic and phonological phenomena, revealing strengths in surface patterns but limitations in applying complex tone rules, and suggesting improvements through linguistic stimuli.

Contribution

It demonstrates that TTS models can capture some linguistic phenomena but struggle with complex tone sandhi, and shows that incorporating BERT embeddings enhances prosody and generalization.

Findings

TTS models capture surface tonal coarticulation well.

Models struggle with Tone-3 sandhi in novel sentences.

BERT embeddings improve naturalness and tone rule generalization.

Abstract

This study probes the phonetic and phonological knowledge of lexical tones in TTS models through two experiments. Controlled stimuli for testing tonal coarticulation and tone sandhi in Mandarin were fed into Tacotron 2 and WaveGlow to generate speech samples, which were subject to acoustic analysis and human evaluation. Results show that both baseline Tacotron 2 and Tacotron 2 with BERT embeddings capture the surface tonal coarticulation patterns well but fail to consistently apply the Tone-3 sandhi rule to novel sentences. Incorporating pre-trained BERT embeddings into Tacotron 2 improves the naturalness and prosody performance, and yields better generalization of Tone-3 sandhi rules to novel complex sentences, although the overall accuracy for Tone-3 sandhi was still low. Given that TTS models do capture some linguistic phenomena, it is argued that they can be used to generate and validate certain linguistic hypotheses. On the other hand, it is also suggested that linguistically informed stimuli should be included in the training and the evaluation of TTS models.