Are Neighbors Enough? Multi-Head Neural n-gram can be Alternative to Self-attention

TL;DR

This paper proposes replacing self-attention in Transformers with a multi-head neural n-gram approach, which considers local context and can match or outperform traditional self-attention in sequence-to-sequence tasks.

Contribution

The study introduces a multi-head neural n-gram mechanism as an alternative to self-attention, demonstrating competitive performance and potential for combined use in Transformers.

Findings

Multi-head neural n-gram achieves comparable or better results than self-attention.

Combining neural n-gram with self-attention further improves performance.

Neural n-gram complements self-attention, enhancing Transformer models.

Abstract

Impressive performance of Transformer has been attributed to self-attention, where dependencies between entire input in a sequence are considered at every position. In this work, we reform the neural $n$-gram model, which focuses on only several surrounding representations of each position, with the multi-head mechanism as in Vaswani et al.(2017). Through experiments on sequence-to-sequence tasks, we show that replacing self-attention in Transformer with multi-head neural $n$-gram can achieve comparable or better performance than Transformer. From various analyses on our proposed method, we find that multi-head neural $n$-gram is complementary to self-attention, and their combinations can further improve performance of vanilla Transformer.