Disentangling Speakers in Multi-Talker Speech Recognition with Speaker-Aware CTC

TL;DR

This paper introduces Speaker-Aware CTC (SACTC), a novel training objective that improves multi-talker speech recognition by explicitly modeling speaker disentanglement, leading to significant word error rate reductions especially in overlapping speech scenarios.

Contribution

It proposes SACTC, a new CTC variant tailored for multi-talker speech recognition, which enhances speaker disentanglement and outperforms existing methods when combined with Serialized Output Training.

Findings

SACTC guides encoder to separate speakers in temporal regions.

Relative WER reductions of 10% overall and 15% on low-overlap speech.

Outperforms standard SOT-CTC in various overlap conditions.

Abstract

Multi-talker speech recognition (MTASR) faces unique challenges in disentangling and transcribing overlapping speech. To address these challenges, this paper investigates the role of Connectionist Temporal Classification (CTC) in speaker disentanglement when incorporated with Serialized Output Training (SOT) for MTASR. Our visualization reveals that CTC guides the encoder to represent different speakers in distinct temporal regions of acoustic embeddings. Leveraging this insight, we propose a novel Speaker-Aware CTC (SACTC) training objective, based on the Bayes risk CTC framework. SACTC is a tailored CTC variant for multi-talker scenarios, it explicitly models speaker disentanglement by constraining the encoder to represent different speakers' tokens at specific time frames. When integrated with SOT, the SOT-SACTC model consistently outperforms standard SOT-CTC across various degrees of speech overlap. Specifically, we observe relative word error rate reductions of 10% overall and 15% on low-overlap speech. This work represents an initial exploration of CTC-based enhancements for MTASR tasks, offering a new perspective on speaker disentanglement in multi-talker speech recognition. The code is available at https://github.com/kjw11/Speaker-Aware-CTC.