Bi-directional Context-Enhanced Speech Large Language Models for Multilingual Conversational ASR

TL;DR

This paper presents a novel multilingual conversational ASR approach that integrates bi-directional context into speech large language models, improving recognition accuracy by 18% through a character-level masking strategy and context-aware decoding.

Contribution

It introduces a context-enhanced speech large language model with a new training and decoding pipeline for multilingual conversational ASR, demonstrating significant performance gains.

Findings

Achieved 18% relative improvement over baseline

Outperformed models trained on larger datasets

Validated on 1500-hour multilingual corpus

Abstract

This paper introduces the integration of language-specific bi-directional context into a speech large language model (SLLM) to improve multilingual continuous conversational automatic speech recognition (ASR). We propose a character-level contextual masking strategy during training, which randomly removes portions of the context to enhance robustness and better emulate the flawed transcriptions that may occur during inference. For decoding, a two-stage pipeline is utilized: initial isolated segment decoding followed by context-aware re-decoding using neighboring hypotheses. Evaluated on the 1500-hour Multilingual Conversational Speech and Language Model (MLC-SLM) corpus covering eleven languages, our method achieves an 18% relative improvement compared to a strong baseline, outperforming even the model trained on 6000 hours of data for the MLC-SLM competition. These results underscore the significant benefit of incorporating contextual information in multilingual continuous conversational ASR.