CALM: Joint Contextual Acoustic-Linguistic Modeling for Personalization of Multi-Speaker ASR

TL;DR

CALM introduces a joint acoustic-linguistic modeling framework for personalized multi-speaker ASR, effectively reducing error rates by integrating speaker cues and contextual biasing in an end-to-end system.

Contribution

It is the first to combine target-speaker extraction with contextual biasing in a unified end-to-end multi-speaker ASR model.

Findings

CALM reduces B-WER from 12.7 to 4.7 on LibriSpeech2Mix.

CALM decreases B-CER from 16.6 to 8.4 on CSJMix2 (eval3).

The framework performs well across English and Japanese speech mixtures.

Abstract

We present CALM, a joint Contextual Acoustic-Linguistic Modeling framework for multi-speaker automatic speech recognition (ASR). In personalized AI scenarios, the joint availability of acoustic and linguistic cues naturally motivates the integration of target-speaker conditioning with contextual biasing in overlapping conversations. CALM implements this integration in an end-to-end framework through speaker embedding-driven target-speaker extraction and dynamic vocabulary-based contextual biasing. We evaluate CALM on simulated English (LibriSpeechMix) and Japanese (Corpus of Spontaneous Japanese mixtures, CSJMix). On two-speaker mixtures, CALM reduces biased word error rate (B-WER) from 12.7 to 4.7 on LibriSpeech2Mix and biased character error rate (B-CER) from 16.6 to 8.4 on CSJMix2 (eval3), demonstrating the effectiveness of joint acoustic-linguistic modeling across languages. We additionally report results on the AMI corpus (IHM-mix condition) to validate performance on standardized speech mixtures.