BUT System Description for CHiME-9 MCoRec Challenge

TL;DR

This paper introduces the BUT system for multi-talker AV-ASR in overlapping conversations, combining a long-context target-speaker model with LLM-based clustering, achieving significant improvements in WER and clustering F1 scores.

Contribution

The work presents a novel AV-ASR architecture conditioned on visual cues and a clustering method using LLMs, advancing multi-talker transcription in complex scenarios.

Findings

Achieved 33.7% WER on MCoRec dev set, 16.2% better than baseline.

Attained a clustering F1 score of 0.97, surpassing previous methods.

Ranked second in the CHiME-9 MCoRec challenge.

Abstract

Multi-talker automatic speech recognition (ASR) in conversational recordings remains an open problem, particularly in scenarios with large portion of overlapping speech where identifying and transcribing a target speaker is difficult from audio alone. Visual cues can help resolve speaker ambiguity, yet their integration into long-context audio-visual (AV) ASR systems has been limited. The CHiME-9 MCoRec task addresses this challenge by requiring transcription of audio-visual recordings of heavily-overlapped parallel conversations, followed by clustering the participants into conversational groups. In this work, we present the BUT system based on a long-context target-speaker AV-ASR model capable of processing long-form recordings in a single decoding pass. Our architecture conditions a pre-trained NVIDIA Parakeet-v2 ASR model on visual representations from a pre-trained AV-HuBERT model. To cluster participants into conversation groups, we employ Qwen3.5-122B LLM to estimate transcript topic similarity followed by hierarchical agglomerative clustering. On the MCoRec development set, the proposed system achieves 33.7% WER and a clustering F1 score of 0.97, improving over the official baseline by 16.2% WER and 0.15 F1 absolute. On the eval set, our team ranked second, being 0.16% WER and 0.5% F1 worse than the best system.