Whisper Finetuning on Nepali Language

TL;DR

This paper demonstrates that fine-tuning OpenAI's Whisper models on a diverse, curated Nepali speech dataset significantly improves transcription accuracy, highlighting the importance of data quality and augmentation for underrepresented languages.

Contribution

It introduces a comprehensive Nepali speech dataset and shows that fine-tuning Whisper models with this data reduces WER and enhances robustness for Nepali ASR.

Findings

WER reduced by up to 36.2% on small models

Data augmentation improves model robustness

Curated dataset outperforms baseline models

Abstract

Despite the growing advancements in Automatic Speech Recognition (ASR) models, the development of robust models for underrepresented languages, such as Nepali, remains a challenge. This research focuses on making an exhaustive and generalized dataset followed by fine-tuning OpenAI's Whisper models of different sizes to improve transcription (speech-to-text) accuracy for the Nepali language. We leverage publicly available ASR datasets and self-recorded custom datasets with a diverse range of accents, dialects, and speaking styles further enriched through augmentation. Our experimental results demonstrate that fine-tuning Whisper models on our curated custom dataset substantially reduces the Word Error Rate (WER) across all model sizes attributed to larger data variations in terms of speaker's age, gender, and sentiment, acoustic environment, dialect, denser audio segments (15-30 seconds) that are more compatible with Whisper's input, and manual curation of audios and transcriptions. Notably, our approach outperforms Whisper's baseline models trained on Fleur's dataset, achieving WER reductions of up to 36.2% on the small and 23.8% on medium models. Furthermore, we show that data augmentation plays a significant role in enhancing model robustness. Our approach underlines the importance of dataset quality, variation, and augmentation in the adaptation of state-of-the-art models to underrepresented languages for developing accurate ASR systems.