A Unified Speech LLM for Diarization and Speech Recognition in Multilingual Conversations

TL;DR

This paper introduces a unified speech LLM capable of jointly performing diarization and speech recognition in multilingual conversations, significantly improving performance on complex conversational tasks.

Contribution

It presents a novel end-to-end model that reformulates training and inference for joint diarization and ASR, addressing ambiguity in pre-segmented audio data.

Findings

Achieved a 54.87% relative improvement in tcpWER/tcpCER over baseline.

Ranked 8th overall in the MLC-SLM Challenge.

Demonstrated effectiveness even with a smaller LLM backbone.

Abstract

Speech Large Language Models (Speech LLMs) have emerged as a crucial paradigm in recent years, extending the capabilities of traditional LLMs to speech tasks such as automatic speech recognition (ASR) and spoken dialogue modeling. However, their effectiveness in real-world multilingual conversations remains limited by the scarcity of data that captures natural conversational phenomena. To address this, the MLC-SLM Challenge provides a multilingual conversational dataset and evaluates models on two tasks: ASR with oracle segmentation (Task I) and joint diarization and recognition without oracle information (Task II). In this paper, we focus on Task II and propose a unified speech LLM that jointly performs diarization and ASR in an end-to-end manner. By reformulating the training data format and modifying the inference procedure, our model addresses the ambiguity inherent in pre-segmented audio and achieves a 54.87\% relative improvement in tcpWER/tcpCER over the baseline, ranking 8th overall, despite using a smaller LLM backbone. We also report results from Task I using a fine-tuned speech LLM.