SC-SOT: Conditioning the Decoder on Diarized Speaker Information for End-to-End Overlapped Speech Recognition

TL;DR

This paper introduces SC-SOT, a novel method that explicitly conditions end-to-end multi-talker speech recognition on speaker information, significantly improving recognition accuracy in overlapped speech scenarios.

Contribution

SC-SOT enhances SOT-based training by integrating speaker embeddings and activity info, explicitly guiding the decoder for better speaker separation in overlapped speech recognition.

Findings

Improved recognition accuracy on overlapped speech datasets.

Effective use of speaker embeddings from diarization models.

Explicit conditioning outperforms implicit separation methods.

Abstract

We propose Speaker-Conditioned Serialized Output Training (SC-SOT), an enhanced SOT-based training for E2E multi-talker ASR. We first probe how SOT handles overlapped speech, and we found the decoder performs implicit speaker separation. We hypothesize this implicit separation is often insufficient due to ambiguous acoustic cues in overlapping regions. To address this, SC-SOT explicitly conditions the decoder on speaker information, providing detailed information about "who spoke when". Specifically, we enhance the decoder by incorporating: (1) speaker embeddings, which allow the model to focus on the acoustic characteristics of the target speaker, and (2) speaker activity information, which guides the model to suppress non-target speakers. The speaker embeddings are derived from a jointly trained E2E speaker diarization model, mitigating the need for speaker enrollment. Experimental results demonstrate the effectiveness of our conditioning approach on overlapped speech.