Analyzing and Fine-Tuning Whisper Models for Multilingual Pilot Speech Transcription in the Cockpit

TL;DR

This study enhances multilingual pilot speech transcription accuracy by applying normalization and fine-tuning Whisper models with Low-Rank Adaptation, significantly reducing Word Error Rate in cockpit speech recognition.

Contribution

It introduces normalization schemes and a fine-tuning approach using LoRA to improve Whisper model performance on niche multilingual cockpit speech data.

Findings

WER reduced from 68.49% to 26.26% with fine-tuning and normalization

Normalization schemes significantly improve transcription accuracy

Fine-tuning with LoRA enhances model adaptation to domain-specific speech

Abstract

The developments in transformer encoder-decoder architectures have led to significant breakthroughs in machine translation, Automatic Speech Recognition (ASR), and instruction-based chat machines, among other applications. The pre-trained models were trained on vast amounts of generic data over a few epochs (fewer than five in most cases), resulting in their strong generalization capabilities. Nevertheless, the performance of these models does suffer when applied to niche domains like transcribing pilot speech in the cockpit, which involves a lot of specific vocabulary and multilingual conversations. This paper investigates and improves the transcription accuracy of cockpit conversations with Whisper models. We have collected around 85 minutes of cockpit simulator recordings and 130 minutes of interview recordings with pilots and manually labeled them. The speakers are middle aged men speaking both German and English. To improve the accuracy of transcriptions, we propose multiple normalization schemes to refine the transcripts and improve Word Error Rate (WER). We then employ fine-tuning to enhance ASR performance, utilizing performance-efficient fine-tuning with Low-Rank Adaptation (LoRA). Hereby, WER decreased from 68.49 \% (pretrained whisper Large model without normalization baseline) to 26.26\% (finetuned whisper Large model with the proposed normalization scheme).