Folding Attention: Memory and Power Optimization for On-Device Transformer-based Streaming Speech Recognition

TL;DR

This paper introduces folding attention, a technique that optimizes memory and power efficiency in on-device Transformer-based streaming speech recognition by reducing model size and power consumption without affecting accuracy.

Contribution

Folding attention specifically targets linear layers in Transformer models, significantly reducing size, memory, and power usage for streaming speech recognition.

Findings

Model size reduced by up to 24%

Power consumption decreased by up to 23%

No loss in model accuracy or computational overhead

Abstract

Transformer-based models excel in speech recognition. Existing efforts to optimize Transformer inference, typically for long-context applications, center on simplifying attention score calculations. However, streaming speech recognition models usually process a limited number of tokens each time, making attention score calculation less of a bottleneck. Instead, the bottleneck lies in the linear projection layers of multi-head attention and feedforward networks, constituting a substantial portion of the model size and contributing significantly to computation, memory, and power usage. To address this bottleneck, we propose folding attention, a technique targeting these linear layers, significantly reducing model size and improving memory and power efficiency. Experiments on on-device Transformer-based streaming speech recognition models show that folding attention reduces model size (and corresponding memory consumption) by up to 24% and power consumption by up to 23%, all without compromising model accuracy or computation overhead.