A Preliminary Analysis of Automatic Word and Syllable Prominence Detection in Non-Native Speech With Text-to-Speech Prosody Embeddings

TL;DR

This study evaluates the effectiveness of prosody embeddings from state-of-the-art TTS systems in detecting word and syllable prominence in native and non-native speech, revealing notable improvements over heuristic and self-supervised features.

Contribution

It introduces a comparative analysis of TTS-derived prosody embeddings for prominence detection in non-native speech, including a novel extraction method during TTS training mode.

Findings

TTS embeddings improve prominence detection accuracy by up to 16.2%.

Embeddings extracted during TTS training mode outperform inference mode.

Prosody embeddings from TTS can match natural speech prominence patterns.

Abstract

Automatic detection of prominence at the word and syllable-levels is critical for building computer-assisted language learning systems. It has been shown that prosody embeddings learned by the current state-of-the-art (SOTA) text-to-speech (TTS) systems could generate word- and syllable-level prominence in the synthesized speech as natural as in native speech. To understand the effectiveness of prosody embeddings from TTS for prominence detection under nonnative context, a comparative analysis is conducted on the embeddings extracted from native and non-native speech considering the prominence-related embeddings: duration, energy, and pitch from a SOTA TTS named FastSpeech2. These embeddings are extracted under two conditions considering: 1) only text, 2) both speech and text. For the first condition, the embeddings are extracted directly from the TTS inference mode, whereas for the second condition, we propose to extract from the TTS under training mode. Experiments are conducted on native speech corpus: Tatoeba, and non-native speech corpus: ISLE. For experimentation, word-level prominence locations are manually annotated for both corpora. The highest relative improvement on word \& syllable-level prominence detection accuracies with the TTS embeddings are found to be 13.7% & 5.9% and 16.2% & 6.9% compared to those with the heuristic-based features and self-supervised Wav2Vec-2.0 representations, respectively.