GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling

TL;DR

GateLoop introduces a data-controlled linear recurrence model that generalizes previous models, offering improved sequence modeling capabilities with efficient modes and providing new insights into attention mechanisms for transformers.

Contribution

The paper develops GateLoop, a novel sequence model that generalizes linear recurrent models with data-controlled state transitions, enhancing performance and theoretical understanding.

Findings

Outperforms existing models in auto-regressive language tasks

Provides efficient $O(l)$ and $O(l \log_{2} l)$ modes for sequence processing

Reveals implications for Transformer architectures through surrogate attention modes

Abstract

Linear Recurrence has proven to be a powerful tool for modeling long sequences efficiently. In this work, we show that existing models fail to take full advantage of its potential. Motivated by this finding, we develop GateLoop, a foundational sequence model that generalizes linear recurrent models such as S4, S5, LRU and RetNet, by employing data-controlled state transitions. Utilizing this theoretical advance, GateLoop empirically outperforms existing models for auto-regressive language modeling. Our method comes with a low-cost $O(l)$ recurrent mode and an efficient $O(l \log_{2} l)$ parallel mode making use of highly optimized associative scan implementations. Furthermore, we derive an $O(l^2)$ surrogate attention mode, revealing remarkable implications for Transformer and recently proposed architectures. Specifically, we prove that our approach can be interpreted as providing data-controlled relative-positional information to Attention. While many existing models solely rely on data-controlled cumulative sums for context aggregation, our findings suggest that incorporating data-controlled complex cumulative products may be a crucial step towards more powerful sequence models.