Fine-tuning Whisper on Low-Resource Languages for Real-World Applications

TL;DR

This paper introduces a novel data generation method for fine-tuning Whisper on low-resource languages, enabling long-form audio transcription and segmentation using only sentence-level data, demonstrated with Swiss German.

Contribution

The paper presents a new data transformation technique that enhances Whisper's performance on low-resource languages without requiring non-sentence-level data.

Findings

Achieved state-of-the-art BLEU scores for Swiss German STT

Improved long-form audio transcription and segmentation capabilities

Method is adaptable to other low-resource languages

Abstract

This paper presents a new approach to fine-tuning OpenAI's Whisper model for low-resource languages by introducing a novel data generation method that converts sentence-level data into a long-form corpus, using Swiss German as a case study. Non-sentence-level data, which could improve the performance of long-form audio, is difficult to obtain and often restricted by copyright laws. Our method bridges this gap by transforming more accessible sentence-level data into a format that preserves the model's ability to handle long-form audio and perform segmentation without requiring non-sentence-level data. Our data generation process improves performance in several real-world applications and leads to the development of a new state-of-the-art speech-to-text (STT) model for Swiss German. We compare our model with a non-fine-tuned Whisper and our previous state-of-the-art Swiss German STT models, where our new model achieves higher BLEU scores. Our results also indicate that the proposed method is adaptable to other low-resource languages, supported by written guidance and code that allows the creation of fine-tuned Whisper models, which keep segmentation capabilities and allow the transcription of longer audio files using only sentence-level data with high quality.