Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddings

TL;DR

This paper introduces a streaming SA-ASR model that jointly transcribes multi-talker speech and recognizes speaker identities with low latency, using token-level speaker embeddings called t-vectors.

Contribution

It proposes a novel encoder-decoder speaker embedding extractor integrated with token-level serialized output training for joint transcription and speaker recognition.

Findings

Achieves better accuracy than previous streaming models.

Performs comparably or better than offline SA-ASR models.

Effective in recognizing speakers in overlapping speech.

Abstract

This paper presents a streaming speaker-attributed automatic speech recognition (SA-ASR) model that can recognize ``who spoke what'' with low latency even when multiple people are speaking simultaneously. Our model is based on token-level serialized output training (t-SOT) which was recently proposed to transcribe multi-talker speech in a streaming fashion. To further recognize speaker identities, we propose an encoder-decoder based speaker embedding extractor that can estimate a speaker representation for each recognized token not only from non-overlapping speech but also from overlapping speech. The proposed speaker embedding, named t-vector, is extracted synchronously with the t-SOT ASR model, enabling joint execution of speaker identification (SID) or speaker diarization (SD) with the multi-talker transcription with low latency. We evaluate the proposed model for a joint task of ASR and SID/SD by using LibriSpeechMix and LibriCSS corpora. The proposed model achieves substantially better accuracy than a prior streaming model and shows comparable or sometimes even superior results to the state-of-the-art offline SA-ASR model.