Streaming Transformer ASR with Blockwise Synchronous Beam Search

TL;DR

This paper introduces a blockwise synchronous beam search algorithm for streaming Transformer-based speech recognition, enabling real-time processing with improved accuracy and reduced latency across multiple languages.

Contribution

The paper presents a novel blockwise processing and alignment method for streaming Transformer ASR, outperforming existing online approaches and enabling effective real-time speech recognition.

Findings

Outperforms conventional online Transformer ASR methods.

Reduces response time through blockwise processing.

Achieves comparable or better accuracy than batch models.

Abstract

The Transformer self-attention network has shown promising performance as an alternative to recurrent neural networks in end-to-end (E2E) automatic speech recognition (ASR) systems. However, Transformer has a drawback in that the entire input sequence is required to compute both self-attention and source--target attention. In this paper, we propose a novel blockwise synchronous beam search algorithm based on blockwise processing of encoder to perform streaming E2E Transformer ASR. In the beam search, encoded feature blocks are synchronously aligned using a block boundary detection technique, where a reliability score of each predicted hypothesis is evaluated based on the end-of-sequence and repeated tokens in the hypothesis. Evaluations of the HKUST and AISHELL-1 Mandarin, LibriSpeech English, and CSJ Japanese tasks show that the proposed streaming Transformer algorithm outperforms conventional online approaches, including monotonic chunkwise attention (MoChA), especially when using the knowledge distillation technique. An ablation study indicates that our streaming approach contributes to reducing the response time, and the repetition criterion contributes significantly in certain tasks. Our streaming ASR models achieve comparable or superior performance to batch models and other streaming-based Transformer methods in all tasks considered.