Speak in Context: Multilingual ASR with Speech Context Alignment via Contrastive Learning

TL;DR

This paper presents a multilingual ASR system that effectively integrates speech context using contrastive learning, enhancing recognition accuracy across diverse languages and dialects while maintaining modularity and flexibility.

Contribution

It introduces a novel contrastive learning approach for aligning speech and contextual representations, enabling improved multilingual and context-aware speech recognition.

Findings

Contextual input improves recognition accuracy by over 5%.

Contrastive alignment enhances performance across different context types.

System supports 11 languages and 5 English dialects with consistent gains.

Abstract

Automatic speech recognition (ASR) has benefited from advances in pretrained speech and language models, yet most systems remain constrained to monolingual settings and short, isolated utterances. While recent efforts in context-aware ASR show promise, two key challenges persist: limited multilingual support and the absence of principled alignment between speech and contextual representations. In this paper, we introduce a context-aware multilingual ASR framework that supports diverse languages and accents while preserving the modularity of pretrained models. Our approach combines a frozen speech encoder and a decoder-only language model via a lightweight projection module, allowing structured context prompts, including dialogue history and biasing words, to guide transcription. To improve interaction between speech and context, we employ a contrastive learning objective that aligns their representations in a shared embedding space. Evaluations on over 1,500 hours of real-world conversational speech across 11 languages and 5 English dialects show that contextual input consistently improves recognition quality. Contrastive alignment provides additional gains when applied to different context types, with an overall performance gain of over 5%. These results highlight the importance of both contextual modeling and cross-modal alignment in multilingual ASR.