How Does Pre-trained Wav2Vec 2.0 Perform on Domain Shifted ASR? An Extensive Benchmark on Air Traffic Control Communications

TL;DR

This paper evaluates the robustness of Wav2Vec 2.0 and XLS-R models on air traffic control speech recognition under domain shift, showing significant improvements over traditional methods and analyzing factors like low-resource performance and gender bias.

Contribution

It provides an extensive benchmark of pre-trained speech models on a new domain, air traffic control, highlighting their robustness and limitations under domain shift conditions.

Findings

Wav2Vec 2.0 and XLS-R outperform hybrid ASR baselines by 20-40% WER reduction.

Fine-tuning with less labeled data achieves substantial performance gains.

Analysis of gender bias and low-resource scenarios reveals model strengths and weaknesses.

Abstract

Recent work on self-supervised pre-training focus on leveraging large-scale unlabeled speech data to build robust end-to-end (E2E) acoustic models (AM) that can be later fine-tuned on downstream tasks e.g., automatic speech recognition (ASR). Yet, few works investigated the impact on performance when the data properties substantially differ between the pre-training and fine-tuning phases, termed domain shift. We target this scenario by analyzing the robustness of Wav2Vec 2.0 and XLS-R models on downstream ASR for a completely unseen domain, air traffic control (ATC) communications. We benchmark these two models on several open-source and challenging ATC databases with signal-to-noise ratio between 5 and 20 dB. Relative word error rate (WER) reductions between 20% to 40% are obtained in comparison to hybrid-based ASR baselines by only fine-tuning E2E acoustic models with a smaller fraction of labeled data. We analyze WERs on the low-resource scenario and gender bias carried by one ATC dataset.