Speaker-Distinguishable CTC: Learning Speaker Distinction Using CTC for Multi-Talker Speech Recognition

TL;DR

This paper introduces Speaker-Distinguishable CTC (SD-CTC), a novel extension of CTC that enables multi-talker speech recognition to distinguish speakers without auxiliary data, improving accuracy and reducing errors.

Contribution

The paper proposes SD-CTC, which jointly assigns tokens and speaker labels, enhancing multi-talker recognition without auxiliary information, integrated into the SOT framework.

Findings

Reduces SOT error rate by 26%

Achieves performance comparable to auxiliary-data methods

Enables speaker distinction using only overlapping speech and transcriptions

Abstract

This paper presents a novel framework for multi-talker automatic speech recognition without the need for auxiliary information. Serialized Output Training (SOT), a widely used approach, suffers from recognition errors due to speaker assignment failures. Although incorporating auxiliary information, such as token-level timestamps, can improve recognition accuracy, extracting such information from natural conversational speech remains challenging. To address this limitation, we propose Speaker-Distinguishable CTC (SD-CTC), an extension of CTC that jointly assigns a token and its corresponding speaker label to each frame. We further integrate SD-CTC into the SOT framework, enabling the SOT model to learn speaker distinction using only overlapping speech and transcriptions. Experimental comparisons show that multi-task learning with SD-CTC and SOT reduces the error rate of the SOT model by 26% and achieves performance comparable to state-of-the-art methods relying on auxiliary information.