In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varieties

TL;DR

This paper demonstrates that in-context learning with few examples significantly improves speech recognition accuracy across diverse speakers and language varieties, especially in low-resource settings, mimicking human listener adaptation.

Contribution

Introduces a scalable in-context learning framework for speech recognition that enhances robustness across speakers and languages, revealing human-like adaptation capabilities in ASR models.

Findings

19.7% relative reduction in word error rate with 12 examples

Most improvements occur in low-resource and matched context scenarios

Scaling context length yields diminishing returns

Abstract

Human listeners readily adjust to unfamiliar speakers and language varieties through exposure, but do these adaptation benefits extend to state-of-the-art spoken language models? We introduce a scalable framework that allows for in-context learning (ICL) in Phi-4 Multimodal using interleaved task prompts and audio-text pairs, and find that as few as 12 example utterances (~50 seconds) at inference time reduce word error rates by a relative 19.7% (1.2 pp.) on average across diverse English corpora. These improvements are most pronounced in low-resource varieties, when the context and target speaker match, and when more examples are provided--though scaling our procedure yields diminishing marginal returns to context length. Overall, we find that our novel ICL adaptation scheme (1) reveals a similar performance profile to human listeners, and (2) demonstrates consistent improvements to automatic speech recognition (ASR) robustness across diverse speakers and language backgrounds. While adaptation succeeds broadly, significant gaps remain for certain varieties, revealing where current models still fall short of human flexibility. We release our prompts and code on GitHub.