Zero-Shot Context-Aware ASR for Diverse Arabic Varieties

TL;DR

This paper introduces a context-aware decoding approach for zero-shot Arabic speech recognition, improving accuracy across diverse dialects and accents by leveraging external information without retraining models.

Contribution

It proposes novel prompt-based and proxy-guided methods for context-aware inference applicable to various ASR architectures, enhancing zero-shot dialectal Arabic recognition.

Findings

Average WER reductions of over 20% on MSA and accented Arabic.

Proxy-guided selection improves WER by 15.6% on MSA.

Context-aware decoding generalizes beyond encoder-decoder models.

Abstract

Zero-shot ASR for Arabic remains challenging: while multilingual models perform well on Modern Standard Arabic (MSA), error rates rise sharply on dialectal and accented speech due to linguistic mismatch and scarce labeled data. We study context-aware decoding as a lightweight test-time adaptation paradigm that conditions inference on external side information without parameter updates. For promptable encoder-decoder ASR (e.g., Whisper), we incorporate context through (i) decoder prompting with first-pass hypotheses and (ii) encoder/decoder prefixing with retrieved speech-text exemplars, complemented by simple prompt reordering and optional speaker-matched synthetic exemplars to improve robustness in informal and multi-speaker settings. To extend contextual adaptation beyond promptable architectures, we introduce proxy-guided n-best selection for CTC ASR: given one or more external proxy hypotheses, we select from a model's n-best list by minimizing text-level distance to the proxies, enabling contextual inference without direct prompting. Across ten Arabic conditions spanning MSA, accented MSA, and multiple dialects, context-aware decoding yields average relative WER reductions of 22.29% on MSA, 20.54 on accented MSA, and 9.15% on dialectal Arabic. For CTC models, proxy-guided selection reduces WER by 15.6% relative on MSA and recovers a substantial fraction of oracle n-best gains, demonstrating that context-aware inference generalizes beyond encoder-decoder ASR.