LearnerVoice: A Dataset of Non-Native English Learners' Spontaneous Speech

TL;DR

This paper introduces LearnerVoice, a new dataset of 50 hours of non-native English learners' spontaneous speech, and demonstrates how fine-tuning ASR models on this data improves recognition accuracy by addressing L2-specific speech features.

Contribution

The paper provides the first publicly available dataset of L2 learner speech with detailed linguistic analysis and shows that fine-tuning improves ASR performance on non-native speech.

Findings

Fine-tuning reduces WER by 44.2% on LearnerVoice.

L2 speech contains significantly more disfluencies and ungrammatical expressions.

Nearly half of the errors in vanilla models are due to L2 speech features.

Abstract

Prevalent ungrammatical expressions and disfluencies in spontaneous speech from second language (L2) learners pose unique challenges to Automatic Speech Recognition (ASR) systems. However, few datasets are tailored to L2 learner speech. We publicly release LearnerVoice, a dataset consisting of 50.04 hours of audio and transcriptions of L2 learners' spontaneous speech. Our linguistic analysis reveals that transcriptions in our dataset contain L2S (L2 learner's Spontaneous speech) features, consisting of ungrammatical expressions and disfluencies (e.g., filler words, word repetitions, self-repairs, false starts), significantly more than native speech datasets. Fine-tuning whisper-small.en with LearnerVoice achieves a WER of 10.26%, 44.2% lower than vanilla whisper-small.en. Furthermore, our qualitative analysis indicates that 54.2% of errors from the vanilla model on LearnerVoice are attributable to L2S features, with 48.1% of them being reduced in the fine-tuned model.