Accurate, fast, cheap: Choose three. Replacing Multi-Head-Attention with Bidirectional Recurrent Attention for Long-Form ASR

TL;DR

This paper introduces bidirectional recurrent attention layers as a more efficient alternative to multi-head attention for long-form speech recognition, achieving comparable accuracy with higher throughput and novel regularization techniques.

Contribution

It demonstrates that bidirectional recurrent attention layers can replace multi-head attention in ASR, offering improved efficiency and introducing Direction Dropout for better regularization and control.

Findings

Bidirectional RA matches MHA accuracy for short and long forms.

RA layers achieve 44% higher throughput than LCA baseline.

Direction Dropout enhances accuracy and throughput control.

Abstract

Long-form speech recognition is an application area of increasing research focus. ASR models based on multi-head attention (MHA) are ill-suited to long-form ASR because of their quadratic complexity in sequence length. We build on recent work that has investigated linear complexity recurrent attention (RA) layers for ASR. We find that bidirectional RA layers can match the accuracy of MHA for both short- and long-form applications. We present a strong limited-context attention (LCA) baseline, and show that RA layers are just as accurate while being more efficient. We develop a long-form training paradigm which further improves RA performance, leading to better accuracy than LCA with 44% higher throughput. We also present Direction Dropout, a novel regularization method that improves accuracy, provides fine-grained control of the accuracy/throughput trade-off of bidirectional RA, and enables a new alternating directions decoding mode with even higher throughput.