Monotonic Chunkwise Attention

TL;DR

This paper introduces Monotonic Chunkwise Attention (MoChA), a method that enables efficient, online, and linear-time decoding in sequence-to-sequence models, improving performance in speech recognition and summarization tasks.

Contribution

MoChA provides a novel attention mechanism that combines monotonicity with chunkwise soft attention, allowing real-time decoding and training with standard backpropagation.

Findings

Achieves state-of-the-art online speech recognition results.

Matches offline soft attention performance in speech tasks.

Significantly improves document summarization over baseline monotonic attention.

Abstract

Sequence-to-sequence models with soft attention have been successfully applied to a wide variety of problems, but their decoding process incurs a quadratic time and space cost and is inapplicable to real-time sequence transduction. To address these issues, we propose Monotonic Chunkwise Attention (MoChA), which adaptively splits the input sequence into small chunks over which soft attention is computed. We show that models utilizing MoChA can be trained efficiently with standard backpropagation while allowing online and linear-time decoding at test time. When applied to online speech recognition, we obtain state-of-the-art results and match the performance of a model using an offline soft attention mechanism. In document summarization experiments where we do not expect monotonic alignments, we show significantly improved performance compared to a baseline monotonic attention-based model.