FusionFormer: Fusing Operations in Transformer for Efficient Streaming Speech Recognition

TL;DR

FusionFormer introduces a normalization and activation strategy in Conformer models that replaces Layer Normalization with Batch Normalization and simpler activations, enabling operator fusion for faster inference without sacrificing accuracy.

Contribution

The paper proposes a novel normalization and activation scheme in Conformer models that reduces inference time by 10% through operator fusion, maintaining performance.

Findings

FusionFormer achieves comparable accuracy to LN-based Conformer.

Inference speed is improved by approximately 10%.

Operator fusion eliminates additional inference costs.

Abstract

The recently proposed Conformer architecture which combines convolution with attention to capture both local and global dependencies has become the \textit{de facto} backbone model for Automatic Speech Recognition~(ASR). Inherited from the Natural Language Processing (NLP) tasks, the architecture takes Layer Normalization~(LN) as a default normalization technique. However, through a series of systematic studies, we find that LN might take 10\% of the inference time despite that it only contributes to 0.1\% of the FLOPs. This motivates us to replace LN with other normalization techniques, e.g., Batch Normalization~(BN), to speed up inference with the help of operator fusion methods and the avoidance of calculating the mean and variance statistics during inference. After examining several plain attempts which directly remove all LN layers or replace them with BN in the same place, we find that the divergence issue is mainly caused by the unstable layer output. We therefore propose to append a BN layer to each linear or convolution layer where stabilized training results are observed. We also propose to simplify the activations in Conformer, such as Swish and GLU, by replacing them with ReLU. All these exchanged modules can be fused into the weights of the adjacent linear/convolution layers and hence have zero inference cost. Therefore, we name it FusionFormer. Our experiments indicate that FusionFormer is as effective as the LN-based Conformer and is about 10\% faster.