SAGE: Spliced-Audio Generated Data for Enhancing Foundational Models in Low-Resource Arabic-English Code-Switched Speech Recognition

TL;DR

This paper introduces SAGE, a data augmentation method using spliced audio to improve low-resource Arabic-English code-switched speech recognition, achieving significant WER reductions and surpassing larger models.

Contribution

The paper presents a novel spliced-audio data generation technique and an experience replay approach to enhance speech recognition in low-resource, dialectal, and code-switched Arabic-English contexts.

Findings

7.8% absolute WER improvement with SAGE data

Outperforms larger multilingual models on CS benchmarks

Reduces WER from 31.7% to 26.6% with language model integration

Abstract

This paper investigates the performance of various speech SSL models on dialectal Arabic (DA) and Arabic-English code-switched (CS) speech. To address data scarcity, a modified audio-splicing approach is introduced to generate artificial CS speech data. Fine-tuning an already fine-tuned SSL model with the proposed Spliced-Audio Generated (SAGE) data results in an absolute improvement on Word Error Rate (WER) of 7.8% on Arabic and English CS benchmarks. Additionally, an Experience Replay (ER) inspired approach is proposed to enhance generalisation across DA and CS speech while mitigating catastrophic forgetting. Integrating an out-of-domain 3-gram language model reduces the overall mean WER from 31.7% to 26.6%. Few-shot fine-tuning for code-switching benchmarks further improves WER by 4.9%. A WER of 31.1% on Arabic-English CS benchmarks surpasses large-scale multilingual models, including USM and Whisper-large-v2 (both over ten times larger) by an absolute margin of 5.5% and 8.4%, respectively.