Enhancing Monotonic Multihead Attention for Streaming ASR

TL;DR

This paper improves streaming speech recognition by enhancing monotonic multihead attention with regularization, head pruning, chunkwise extension, and synchronized decoding, leading to more reliable online transcription.

Contribution

It introduces HeadDrop regularization, head pruning, chunkwise multihead extension, and head-synchronous decoding to improve monotonic multihead attention in streaming ASR.

Findings

All MA heads learn proper alignments with proposed regularization.

Pruning redundant heads enhances boundary detection accuracy.

The method achieves stable and accurate streaming ASR performance.

Abstract

We investigate a monotonic multihead attention (MMA) by extending hard monotonic attention to Transformer-based automatic speech recognition (ASR) for online streaming applications. For streaming inference, all monotonic attention (MA) heads should learn proper alignments because the next token is not generated until all heads detect the corresponding token boundaries. However, we found not all MA heads learn alignments with a na\"ive implementation. To encourage every head to learn alignments properly, we propose HeadDrop regularization by masking out a part of heads stochastically during training. Furthermore, we propose to prune redundant heads to improve consensus among heads for boundary detection and prevent delayed token generation caused by such heads. Chunkwise attention on each MA head is extended to the multihead counterpart. Finally, we propose head-synchronous beam search decoding to guarantee stable streaming inference.