Pointer: Linear-Complexity Long-Range Modeling without Pre-training

TL;DR

Pointer introduces a linear-complexity architecture for long-range sequence modeling that outperforms standard transformers in speed and maintains high accuracy without pre-training, using explicit pointer chains for dependencies.

Contribution

It proposes a novel pointer-based architecture achieving linear complexity for long-range modeling without pre-training, with interpretable pointer patterns and significant speed improvements.

Findings

2-10x speedup on long sequences

>95% accuracy on copy tasks up to 2048 tokens

Learned pointer patterns reveal structured dependencies

Abstract

We introduce Pointer, a novel architecture that achieves linear $O(NK)$ complexity for long-range sequence modeling while maintaining superior performance without requiring pre-training. Unlike standard attention mechanisms that compute $O(N^2)$ pairwise interactions, our approach uses layer-wise pointer chaining where each layer's pointer selection depends on previous layer's pointer positions, creating explicit long-distance connections through pointer chains. We demonstrate that this architecture achieves $2$--$10\times$ speedup on long sequences compared to standard transformers, maintains $>95\%$ accuracy on copy tasks at distances up to 2048 tokens, and learns interpretable pointer patterns that reveal structured dependency modeling. Our experiments on efficiency benchmarks, long-range dependency tasks, and interpretability analysis show that Pointer offers a compelling alternative to attention mechanisms for scenarios requiring efficient long-range modeling without pre-training dependencies.