The DKU System for Multi-Speaker Automatic Speech Recognition in MLC-SLM Challenge

TL;DR

The paper introduces a multi-speaker speech recognition system that integrates diarization and language modeling, achieving state-of-the-art results on the MLC-SLM dataset without relying on oracle labels.

Contribution

It presents a novel diarization-aware LLM framework with multilingual fine-tuning, improving multi-speaker ASR performance significantly.

Findings

Achieved tcpWER of 23.56% on development set

Achieved tcpWER of 18.08% on test set

Outperformed the official baseline substantially

Abstract

We present the DKU system for Task 2 of the MLC-SLM Challenge, which aims to perform multi-speaker automatic speech recognition directly from raw audio without Oracle speaker labels or time boundaries. Our approach builds upon a diarization-aware framework integrating speaker embeddings and temporal utterance boundaries into a Qwen2.5-based large language model (LLM). Then, we enhance the system's multilingual performance by fine-tuning language-specific adapters and LoRA modules within the LLM decoder. Finally, our system achieves the tcpWER of 23.56\% and 18.08\% on the development and test sets of the MLC-SLM dataset, substantially outperforming the official baseline.