Attention Enhanced Citrinet for Speech Recognition

TL;DR

This paper introduces an attention-enhanced Citrinet model for speech recognition that converges faster and achieves lower error rates by integrating multi-head attention and reducing layers, outperforming previous models in speed and accuracy.

Contribution

The paper proposes a novel modification to Citrinet by adding multi-head attention and reducing layers, resulting in faster convergence and improved performance.

Findings

Faster convergence compared to original Citrinet.

Lower character error rates on Japanese datasets.

Reduced model size and training time.

Abstract

Citrinet is an end-to-end convolutional Connectionist Temporal Classification (CTC) based automatic speech recognition (ASR) model. To capture local and global contextual information, 1D time-channel separable convolutions combined with sub-word encoding and squeeze-and-excitation (SE) are used in Citrinet, making the whole architecture to be as deep as including 23 blocks with 235 convolution layers and 46 linear layers. This pure convolutional and deep architecture makes Critrinet relatively slow at convergence. In this paper, we propose to introduce multi-head attentions together with feed-forward networks in the convolution module in Citrinet blocks while keeping the SE module and residual module unchanged. For speeding up, we remove 8 convolution layers in each attention-enhanced Citrinet block and reduce 23 blocks to 13. Experiments on the Japanese CSJ-500h and Magic-1600h dataset show that the attention-enhanced Citrinet with less layers and blocks and converges faster with lower character error rates than (1) Citrinet with 80\% training time and (2) Conformer with 40\% training time and 29.8\% model size.