Token-Level Serialized Output Training for Joint Streaming ASR and ST Leveraging Textual Alignments

TL;DR

This paper presents a streaming Transformer-Transducer model that jointly produces speech recognition and translation outputs with minimal latency, leveraging token-level serialized training and textual alignments to improve quality and efficiency.

Contribution

It introduces a novel joint token-level serialized output training method for streaming ASR and ST, enabling simultaneous generation with low latency and improved accuracy.

Findings

Achieves a latency of 1s for ASR and 1.3s for ST.

Improves WER by 1.1 and BLEU by 0.4 in multilingual settings.

Maintains or enhances output quality compared to separate models.

Abstract

In real-world applications, users often require both translations and transcriptions of speech to enhance their comprehension, particularly in streaming scenarios where incremental generation is necessary. This paper introduces a streaming Transformer-Transducer that jointly generates automatic speech recognition (ASR) and speech translation (ST) outputs using a single decoder. To produce ASR and ST content effectively with minimal latency, we propose a joint token-level serialized output training method that interleaves source and target words by leveraging an off-the-shelf textual aligner. Experiments in monolingual (it-en) and multilingual (\{de,es,it\}-en) settings demonstrate that our approach achieves the best quality-latency balance. With an average ASR latency of 1s and ST latency of 1.3s, our model shows no degradation or even improves output quality compared to separate ASR and ST models, yielding an average improvement of 1.1 WER and 0.4 BLEU in the multilingual case.