BUT System for the MLC-SLM Challenge

TL;DR

This paper introduces a two-speaker ASR system combining diarization and speech recognition models, demonstrating strong out-of-domain generalization, effective fine-tuning, and addressing data quality issues to improve performance in multilingual scenarios.

Contribution

The paper presents a novel combination of DiCoW and DiariZen for two-speaker ASR, showing their robustness and effectiveness in multilingual and domain-adapted settings, and highlights data labeling issues affecting diarization.

Findings

DiariZen outperforms Pyannote in out-of-domain scenarios.

Fine-tuned DiCoW maintains multilingual performance.

System achieves 16.75% tcpWER/CER, ranking second in the challenge.

Abstract

We present a two-speaker automatic speech recognition (ASR) system that combines DiCoW -- a diarization-conditioned variant of Whisper -- with DiariZen, a diarization pipeline built on top of Pyannote. We first evaluate both systems in out-of-domain (OOD) multilingual scenarios without any fine-tuning. In this scenario, DiariZen consistently outperforms the baseline Pyannote diarization model, demonstrating strong generalization. Despite being fine-tuned on English-only data for target-speaker ASR, DiCoW retains solid multilingual performance, indicating that encoder modifications preserve Whisper's multilingual capabilities. We then fine-tune both DiCoW and DiariZen on the MLC-SLM challenge data. The fine-tuned DiariZen continues to outperform the fine-tuned Pyannote baseline, while DiCoW sees further gains from domain adaptation. Our final system achieves a micro-average tcpWER/CER of 16.75% and ranks second in Task 2 of the MLC-SLM challenge. Lastly, we identify several labeling inconsistencies in the training data -- such as missing speech segments and incorrect silence annotations -- which can hinder diarization fine-tuning. We propose simple mitigation strategies to address these issues and improve system robustness.