Mispronunciation detection using self-supervised speech representations

TL;DR

This paper explores the use of self-supervised speech representations for detecting mispronunciations in second language learners, comparing different training strategies and representations on non-native speech datasets.

Contribution

It demonstrates that training a downstream model directly for mispronunciation detection yields better results than phone recognition training, with most SSL representations performing similarly.

Findings

Targeted training improves detection accuracy.

Most SSL representations perform comparably.

Traditional DNN features are less effective.

Abstract

In recent years, self-supervised learning (SSL) models have produced promising results in a variety of speech-processing tasks, especially in contexts of data scarcity. In this paper, we study the use of SSL models for the task of mispronunciation detection for second language learners. We compare two downstream approaches: 1) training the model for phone recognition (PR) using native English data, and 2) training a model directly for the target task using non-native English data. We compare the performance of these two approaches for various SSL representations as well as a representation extracted from a traditional DNN-based speech recognition model. We evaluate the models on L2Arctic and EpaDB, two datasets of non-native speech annotated with pronunciation labels at the phone level. Overall, we find that using a downstream model trained for the target task gives the best performance and that most upstream models perform similarly for the task.