Efficient Context Propagating Perceiver Architectures for Auto-Regressive Language Modeling

TL;DR

This paper introduces the Efficient Context propagating Perceiver (ECP), a novel architecture that improves long-sequence language modeling by reducing attention complexity while maintaining high performance, outperforming existing models on multiple benchmarks.

Contribution

The paper develops four new Perceiver-based architectures, with ECP being the best, which overcomes key limitations of prior models by combining context and latent sequences efficiently.

Findings

ECP significantly outperforms state-of-the-art Transformer models on Wikitext-103.

ECP operates with the same attention complexity as LongLoRA, ensuring computational efficiency.

ECP achieves better language modeling performance through pairwise segment attention.

Abstract

One of the key challenges in Transformer architectures is the quadratic complexity of the attention mechanism, which limits the efficient processing of long sequences. Many recent research works have attempted to provide a reduction from the $O(n^2)$ time complexity of attention to semi-linear complexity. However, it remains an unsolved problem in the sense of maintaining high performance when complexity is reduced. One of the important works in this respect is the Perceiver class of architectures that have demonstrated excellent performance, while reducing the computation complexity. In this paper, we use the PerceiverAR as a basis and explore the design space of different trade-offs between preserving context and reducing attention complexity. To this end, we develop four new architectural paradigms, the best performing of which we denote as the Efficient Context propagating Perceiver (ECP). ECP has two major advantages over the PerceiverAR. First, the ECP architecture overcomes the main drawback of PercieverAR by utilizing both the context and the latent sequences in autoregressive training. Second, the ECP architecture operates with the same attention complexity as LongLoRA, making it computationally efficient. More importantly, via pairwise segment attention, it extracts better information resulting in improved language modeling. Empirically, we demonstrate that the ECP architecture significantly outperforms other state-of-the-art Transformer models on Wikitext-103, PG-19 and sCIFAR-10.