Shifted Chunk Encoder for Transformer Based Streaming End-to-End ASR

TL;DR

This paper introduces a shifted chunk mechanism for Transformer-based streaming ASR, enhancing global context modeling and efficiency while maintaining the advantages of chunk-wise approaches.

Contribution

It proposes a novel shifted chunk mechanism that improves global context modeling in chunk-wise Transformer models for streaming ASR.

Findings

Achieves CER of 6.43% with SChunk-Transformer and 5.77% with SChunk-Conformer on AISHELL-1.

Models have linear complexity, enabling efficient training and inference.

Outperforms conventional chunk-wise models and is competitive with memory-based methods.

Abstract

Currently, there are mainly three kinds of Transformer encoder based streaming End to End (E2E) Automatic Speech Recognition (ASR) approaches, namely time-restricted methods, chunk-wise methods, and memory-based methods. Generally, all of them have limitations in aspects of linear computational complexity, global context modeling, and parallel training. In this work, we aim to build a model to take all these three advantages for streaming Transformer ASR. Particularly, we propose a shifted chunk mechanism for the chunk-wise Transformer which provides cross-chunk connections between chunks. Therefore, the global context modeling ability of chunk-wise models can be significantly enhanced while all the original merits inherited. We integrate this scheme with the chunk-wise Transformer and Conformer, and identify them as SChunk-Transformer and SChunk-Conformer, respectively. Experiments on AISHELL-1 show that the SChunk-Transformer and SChunk-Conformer can respectively achieve CER 6.43% and 5.77%. And the linear complexity makes them possible to train with large batches and infer more efficiently. Our models can significantly outperform their conventional chunk-wise counterparts, while being competitive, with only 0.22 absolute CER drop, when compared with U2 which has quadratic complexity. A better CER can be achieved if compared with existing chunk-wise or memory-based methods, such as HS-DACS and MMA. Code is released.