Pitch-Aware RNN-T for Mandarin Chinese Mispronunciation Detection and Diagnosis

TL;DR

This paper proposes a pitch-aware RNN-T model for Mandarin Chinese mispronunciation detection that improves accuracy by integrating pitch features and addressing dataset limitations.

Contribution

It introduces a novel stateless RNN-T architecture with pitch embedding for Mandarin MDD, reducing information gaps and enhancing detection performance.

Findings

3% reduction in Phone Error Rate

7% increase in False Acceptance Rate

Effective training solely on native speaker data

Abstract

Mispronunciation Detection and Diagnosis (MDD) systems, leveraging Automatic Speech Recognition (ASR), face two main challenges in Mandarin Chinese: 1) The two-stage models create an information gap between the phoneme or tone classification stage and the MDD stage. 2) The scarcity of Mandarin MDD datasets limits model training. In this paper, we introduce a stateless RNN-T model for Mandarin MDD, utilizing HuBERT features with pitch embedding through a Pitch Fusion Block. Our model, trained solely on native speaker data, shows a 3% improvement in Phone Error Rate and a 7% increase in False Acceptance Rate over the state-of-the-art baseline in non-native scenarios