Squeezeformer: An Efficient Transformer for Automatic Speech Recognition

TL;DR

Squeezeformer introduces a simplified, efficient transformer architecture for speech recognition that outperforms the Conformer model by optimizing macro and micro-architecture components, achieving state-of-the-art WERs.

Contribution

The paper proposes Squeezeformer, a novel architecture that improves upon Conformer by simplifying design choices and incorporating efficient modules, leading to better performance in ASR tasks.

Findings

Squeezeformer achieves 6.0-7.5% WER on LibriSpeech test sets.

It outperforms Conformer-CTC with the same FLOPs by 0.6-3.1% WER.

The model is more efficient due to the Temporal U-Net and depthwise down-sampling layers.

Abstract

The recently proposed Conformer model has become the de facto backbone model for various downstream speech tasks based on its hybrid attention-convolution architecture that captures both local and global features. However, through a series of systematic studies, we find that the Conformer architecture's design choices are not optimal. After re-examining the design choices for both the macro and micro-architecture of Conformer, we propose Squeezeformer which consistently outperforms the state-of-the-art ASR models under the same training schemes. In particular, for the macro-architecture, Squeezeformer incorporates (i) the Temporal U-Net structure which reduces the cost of the multi-head attention modules on long sequences, and (ii) a simpler block structure of multi-head attention or convolution modules followed up by feed-forward module instead of the Macaron structure proposed in Conformer. Furthermore, for the micro-architecture, Squeezeformer (i) simplifies the activations in the convolutional block, (ii) removes redundant Layer Normalization operations, and (iii) incorporates an efficient depthwise down-sampling layer to efficiently sub-sample the input signal. Squeezeformer achieves state-of-the-art results of 7.5%, 6.5%, and 6.0% word-error-rate (WER) on LibriSpeech test-other without external language models, which are 3.1%, 1.4%, and 0.6% better than Conformer-CTC with the same number of FLOPs. Our code is open-sourced and available online.