Streaming End-to-End ASR based on Blockwise Non-Autoregressive Models

TL;DR

This paper introduces a novel streaming non-autoregressive end-to-end speech recognition system that processes audio in small blocks, achieving low latency and faster inference while maintaining accuracy.

Contribution

It combines blockwise attention and Mask-CTC with an overlapping decoding strategy for real-time speech recognition, addressing latency and coherence issues.

Findings

Improved online ASR accuracy in low latency scenarios

Faster inference speed compared to autoregressive models

Effective handling of edge errors with overlapping decoding

Abstract

Non-autoregressive (NAR) modeling has gained more and more attention in speech processing. With recent state-of-the-art attention-based automatic speech recognition (ASR) structure, NAR can realize promising real-time factor (RTF) improvement with only small degradation of accuracy compared to the autoregressive (AR) models. However, the recognition inference needs to wait for the completion of a full speech utterance, which limits their applications on low latency scenarios. To address this issue, we propose a novel end-to-end streaming NAR speech recognition system by combining blockwise-attention and connectionist temporal classification with mask-predict (Mask-CTC) NAR. During inference, the input audio is separated into small blocks and then processed in a blockwise streaming way. To address the insertion and deletion error at the edge of the output of each block, we apply an overlapping decoding strategy with a dynamic mapping trick that can produce more coherent sentences. Experimental results show that the proposed method improves online ASR recognition in low latency conditions compared to vanilla Mask-CTC. Moreover, it can achieve a much faster inference speed compared to the AR attention-based models. All of our codes will be publicly available at https://github.com/espnet/espnet.