DiariST: Streaming Speech Translation with Speaker Diarization

TL;DR

DiariST is a pioneering streaming speech translation and speaker diarization system that handles overlapping speech in real-time, supported by a new dataset and evaluation metrics for this challenging task.

Contribution

It introduces the first streaming solution for speech translation with speaker diarization, along with a new dataset and metrics for evaluating such systems.

Findings

Achieves strong streaming speech translation and diarization performance.

Handles overlapping speech effectively in streaming mode.

Provides new benchmarks and tools for future research.

Abstract

End-to-end speech translation (ST) for conversation recordings involves several under-explored challenges such as speaker diarization (SD) without accurate word time stamps and handling of overlapping speech in a streaming fashion. In this work, we propose DiariST, the first streaming ST and SD solution. It is built upon a neural transducer-based streaming ST system and integrates token-level serialized output training and t-vector, which were originally developed for multi-talker speech recognition. Due to the absence of evaluation benchmarks in this area, we develop a new evaluation dataset, DiariST-AliMeeting, by translating the reference Chinese transcriptions of the AliMeeting corpus into English. We also propose new metrics, called speaker-agnostic BLEU and speaker-attributed BLEU, to measure the ST quality while taking SD accuracy into account. Our system achieves a strong ST and SD capability compared to offline systems based on Whisper, while performing streaming inference for overlapping speech. To facilitate the research in this new direction, we release the evaluation data, the offline baseline systems, and the evaluation code.