Modality Agnostic Efficient Long Range Encoder

TL;DR

This paper introduces MAELRE, a modality-agnostic transformer architecture that efficiently processes long-range sequences on a single device by reducing quadratic complexity through token merging and attention approximation, achieving better accuracy and efficiency.

Contribution

MAELRE is a novel unified transformer architecture that combines token merging with attention approximation for efficient long-range encoding across multiple modalities.

Findings

MAELRE outperforms existing models in accuracy on diverse modality tasks.

It significantly reduces computational cost compared to traditional long-context models.

MAELRE maintains high accuracy while handling longer sequences efficiently.

Abstract

The long-context capability of recent large transformer models can be surmised to rely on techniques such as attention/model parallelism, as well as hardware-level optimizations. While these strategies allow input lengths to scale to millions of tokens, they do not fundamentally mitigate the quadratic computational and memory complexity of the core attention mechanism. In this paper, we address the challenge of long-context processing on a single device using generic implementations by reducing the quadratic memory footprint and inference cost. Existing approaches to extend the context length for generic single device implementations -- such as token merging and modified attentions -- are often modality specific and attain a suboptimal tradeoff between accuracy and efficiency. To overcome these limitations, we propose MAELRE (Modality Agnostic Efficient Long Range Encoder), a unified and efficient transformer architecture designed for long-range encoding across diverse modalities. MAELRE integrates token merging with attention approximation, progressively merging tokens at different stages of internal computational blocks. It employs a lightweight attention approximation when the number of tokens is large, and switches to standard dot-product attention as the sequence becomes shorter through successive aggregation. We demonstrate that MAELRE achieves superior accuracy while reducing computational cost compared to existing long-context models on classification tasks spanning multiple modalities, including text, time series, audio, and vision.