t-SOT FNT: Streaming Multi-talker ASR with Text-only Domain Adaptation Capability

TL;DR

This paper introduces a novel t-SOT FNT model for streaming multi-talker ASR that effectively separates language modeling from the transducer, enabling text-only domain adaptation while maintaining high recognition accuracy.

Contribution

It proposes a factorized neural transducer architecture for t-SOT, improving text-only adaptation capabilities in multi-talker speech recognition.

Findings

Achieves comparable WER to original t-SOT model

Enables effective text-only domain adaptation

Reduces word error rate on multi-talker datasets

Abstract

Token-level serialized output training (t-SOT) was recently proposed to address the challenge of streaming multi-talker automatic speech recognition (ASR). T-SOT effectively handles overlapped speech by representing multi-talker transcriptions as a single token stream with $\langle \text{cc}\rangle$ symbols interspersed. However, the use of a naive neural transducer architecture significantly constrained its applicability for text-only adaptation. To overcome this limitation, we propose a novel t-SOT model structure that incorporates the idea of factorized neural transducers (FNT). The proposed method separates a language model (LM) from the transducer's predictor and handles the unnatural token order resulting from the use of $\langle \text{cc}\rangle$ symbols in t-SOT. We achieve this by maintaining multiple hidden states and introducing special handling of the $\langle \text{cc}\rangle$ tokens within the LM. The proposed t-SOT FNT model achieves comparable performance to the original t-SOT model while retaining the ability to reduce word error rate (WER) on both single and multi-talker datasets through text-only adaptation.