Improving the Inclusivity of Dutch Speech Recognition by Fine-tuning Whisper on the JASMIN-CGN Corpus

TL;DR

This study evaluates how fine-tuning Whisper on specific Dutch speech subpopulations improves recognition accuracy for children, elderly, and non-native speakers, highlighting the importance of inclusive training data.

Contribution

It demonstrates significant WER reductions by fine-tuning Whisper on underrepresented Dutch speech groups, emphasizing tailored model adaptation for inclusivity.

Findings

Fine-tuning reduces WER by up to 81% for native children.

Performance improvements are substantial across all targeted subpopulations.

Training on specific groups enhances recognition accuracy for diverse speakers.

Abstract

We test and study the variation in speech recognition of fine-tuned versions of the Whisper model on child, elderly and non-native Dutch speech from the JASMIN-CGN corpus. Our primary goal is to evaluate how speakers' age and linguistic background influence Whisper's performance. Whisper achieves varying Word Error Rates (WER) when fine-tuned on subpopulations of specific ages and linguistic backgrounds. Fine-tuned performance is remarkably better than zero-shot performance, achieving a relative reduction in WER of 81% for native children, 72% for non-native children, 67% for non-native adults, and 65% for native elderly people. Our findings underscore the importance of training speech recognition models like Whisper on underrepresented subpopulations such as children, the elderly, and non-native speakers.