Dispatcher: A Message-Passing Approach To Language Modelling

TL;DR

This paper introduces the Dispatcher message-passing layer for language modeling, replacing self-attention to improve efficiency while maintaining competitive perplexity, with lower computational and memory complexity.

Contribution

It presents a novel message-passing layer that substitutes self-attention in language models, achieving efficiency gains without sacrificing performance.

Findings

Achieves O(N logN) computational complexity

Maintains comparable perplexity to existing methods

Uses O(N) memory complexity

Abstract

This paper proposes a message-passing mechanism to address language modelling. A new layer type is introduced that aims to substitute self-attention for unidirectional sequence generation tasks. The system is shown to be competitive with existing methods: Given N tokens, the computational complexity is O(N logN) and the memory complexity is O(N) under reasonable assumptions. In the end, the Dispatcher layer is seen to achieve comparable perplexity to prior results while being more efficient.