Improving Autoregressive NLP Tasks via Modular Linearized Attention

TL;DR

This paper introduces modular linearized attention (MLA), a novel approach combining efficient attention mechanisms to improve inference speed and maintain performance in autoregressive NLP tasks, especially in resource-constrained settings.

Contribution

The paper presents MLA, a new attention mechanism that enhances efficiency and performance in autoregressive NLP models, validated across multiple tasks.

Findings

Notable speedups in inference time.

Maintains competitive performance in NMT and SimulST.

Efficiency gains demonstrated in TTS applications.

Abstract

Various natural language processing (NLP) tasks necessitate models that are efficient and small based on their ultimate application at the edge or in other resource-constrained environments. While prior research has reduced the size of these models, increasing computational efficiency without considerable performance impacts remains difficult, especially for autoregressive tasks. This paper proposes modular linearized attention (MLA), which combines multiple efficient attention mechanisms, including cosFormer, to maximize inference quality while achieving notable speedups. We validate this approach on several autoregressive NLP tasks, including speech-to-text neural machine translation (S2T NMT), speech-to-text simultaneous translation (SimulST), and autoregressive text-to-spectrogram, noting efficiency gains on TTS and competitive performance for NMT and SimulST during training and inference.