Fast-U2++: Fast and Accurate End-to-End Speech Recognition in Joint CTC/Attention Frames

TL;DR

Fast-U2++ is an improved end-to-end speech recognition model that significantly reduces latency while maintaining high accuracy by using multi-chunk processing and knowledge distillation, demonstrated on the Aishell-1 dataset.

Contribution

It introduces fast-U2++, a novel model that reduces latency through multi-chunk encoder processing and knowledge distillation, enhancing the original U2++ framework.

Findings

Reduces model latency from 320ms to 80ms.

Achieves a CER of 5.06% on Aishell-1.

Maintains high accuracy with low latency.

Abstract

Recently, the unified streaming and non-streaming two-pass (U2/U2++) end-to-end model for speech recognition has shown great performance in terms of streaming capability, accuracy and latency. In this paper, we present fast-U2++, an enhanced version of U2++ to further reduce partial latency. The core idea of fast-U2++ is to output partial results of the bottom layers in its encoder with a small chunk, while using a large chunk in the top layers of its encoder to compensate the performance degradation caused by the small chunk. Moreover, we use knowledge distillation method to reduce the token emission latency. We present extensive experiments on Aishell-1 dataset. Experiments and ablation studies show that compared to U2++, fast-U2++ reduces model latency from 320ms to 80ms, and achieves a character error rate (CER) of 5.06% with a streaming setup.