Capturing Multi-Resolution Context by Dilated Self-Attention

TL;DR

This paper introduces dilated self-attention, combining restricted attention and dilation to efficiently capture multi-resolution context in long sequences, significantly reducing computational costs while maintaining high performance.

Contribution

It proposes a novel dilated self-attention mechanism that balances local and distant information capture with lower computational complexity.

Findings

Achieves ASR performance comparable to full self-attention.

Reduces computational costs significantly.

Demonstrates effectiveness of dilation methods like pooling and subsampling.

Abstract

Self-attention has become an important and widely used neural network component that helped to establish new state-of-the-art results for various applications, such as machine translation and automatic speech recognition (ASR). However, the computational complexity of self-attention grows quadratically with the input sequence length. This can be particularly problematic for applications such as ASR, where an input sequence generated from an utterance can be relatively long. In this work, we propose a combination of restricted self-attention and a dilation mechanism, which we refer to as dilated self-attention. The restricted self-attention allows attention to neighboring frames of the query at a high resolution, and the dilation mechanism summarizes distant information to allow attending to it with a lower resolution. Different methods for summarizing distant frames are studied, such as subsampling, mean-pooling, and attention-based pooling. ASR results demonstrate substantial improvements compared to restricted self-attention alone, achieving similar results compared to full-sequence based self-attention with a fraction of the computational costs.