Improving Child Speech Recognition and Reading Mistake Detection by Using Prompts

TL;DR

This paper introduces a multimodal approach using prompts with Whisper and large language models to enhance child speech recognition and reading mistake detection, achieving state-of-the-art results in Dutch.

Contribution

It demonstrates the effectiveness of prompt-based methods with audio and text models for improving child speech recognition and error detection.

Findings

Achieved a WER of 5.1% in Dutch child speech recognition.

Increased reading mistake detection F1 score from 0.39 to 0.73.

Outperformed baseline models with prompt-based techniques.

Abstract

Automatic reading aloud evaluation can provide valuable support to teachers by enabling more efficient scoring of reading exercises. However, research on reading evaluation systems and applications remains limited. We present a novel multimodal approach that leverages audio and knowledge from text resources. In particular, we explored the potential of using Whisper and instruction-tuned large language models (LLMs) with prompts to improve transcriptions for child speech recognition, as well as their effectiveness in downstream reading mistake detection. Our results demonstrate the effectiveness of prompting Whisper and prompting LLM, compared to the baseline Whisper model without prompting. The best performing system achieved state-of-the-art recognition performance in Dutch child read speech, with a word error rate (WER) of 5.1%, improving the baseline WER of 9.4%. Furthermore, it significantly improved reading mistake detection, increasing the F1 score from 0.39 to 0.73.