Automatic Pronunciation Assessment using Self-Supervised Speech Representation Learning

TL;DR

This paper introduces a novel SSL-based approach for automatic pronunciation assessment that fine-tunes pre-trained models and extracts layer-wise representations to accurately evaluate ESL learners' pronunciation.

Contribution

It presents a new SSL model fine-tuning method combined with layer-wise representation extraction for improved pronunciation scoring in ESL contexts.

Findings

Outperforms baseline methods in Pearson correlation coefficient

Effective on datasets of Korean ESL children and Speechocean762

Analyzes impact of different transformer layer representations

Abstract

Self-supervised learning (SSL) approaches such as wav2vec 2.0 and HuBERT models have shown promising results in various downstream tasks in the speech community. In particular, speech representations learned by SSL models have been shown to be effective for encoding various speech-related characteristics. In this context, we propose a novel automatic pronunciation assessment method based on SSL models. First, the proposed method fine-tunes the pre-trained SSL models with connectionist temporal classification to adapt the English pronunciation of English-as-a-second-language (ESL) learners in a data environment. Then, the layer-wise contextual representations are extracted from all across the transformer layers of the SSL models. Finally, the automatic pronunciation score is estimated using bidirectional long short-term memory with the layer-wise contextual representations and the corresponding text. We show that the proposed SSL model-based methods outperform the baselines, in terms of the Pearson correlation coefficient, on datasets of Korean ESL learner children and Speechocean762. Furthermore, we analyze how different representations of transformer layers in the SSL model affect the performance of the pronunciation assessment task.