Bridging the gap: A comparative exploration of Speech-LLM and end-to-end architecture for multilingual conversational ASR

TL;DR

This paper explores and compares speech-language models and end-to-end architectures for multilingual conversational speech recognition, proposing an enhanced LLM-based system that achieves competitive results with less training data.

Contribution

It introduces an improved LLM-based ASR framework combining fine-tuned Whisper and mHuBERT encoders with novel cross-attention fusion mechanisms, and evaluates their performance against end-to-end models.

Findings

The proposed system achieves 10.69% CER/WER, ranking among top systems.

Fine-tuned Whisper models with LoRA perform well on multilingual ASR.

LLM-based ASR still lags behind fine-tuned end-to-end Whisper models.

Abstract

The INTERSPEECH 2025 Challenge on Multilingual Conversational Speech Language Models (MLC-SLM) promotes multilingual conversational ASR with large language models (LLMs). Our previous SHNU-mASR system adopted a competitive parallel-speech-encoder architecture that integrated Whisper and mHuBERT with an LLM. However, it faced two challenges: simple feature concatenation may not fully exploit complementary information, and the performance gap between LLM-based ASR and end-to-end(E2E) encoder-decoder ASR remained unexplored. In this work, we present an enhanced LLM-based ASR framework that combines fine-tuned Whisper and mHuBERT encoders with an LLM to enrich speech representations. We first evaluate E2E Whisper models with LoRA and full fine-tuning on the MLC-SLM ASR task, and then propose cross-attention-based fusion mechanisms for the parallel-speech-encoder. On the official evaluation set of the MLC-SLM Challenge, our system achieves a CER/WER of 10.69%, ranking on par with the top-ranked Track 1 systems, even though it uses only 1,500 hours of baseline training data compared with their large-scale training sets. Nonetheless, we find that our final LLM-based ASR still does not match the performance of a fine-tuned E2E Whisper model, providing valuable empirical guidance for future Speech-LLM design. Our code is publicly available at https://github.com/1535176727/MLC-SLM.