Learning When Not to Attend Globally

TL;DR

This paper introduces All-or-Here Attention (AHA), a method enabling Large Language Models to selectively switch between full and local attention, significantly reducing computational costs while maintaining performance.

Contribution

AHA is a novel attention mechanism that dynamically toggles between full and local attention, optimizing efficiency in LLMs based on context needs.

Findings

Up to 93% of full attention operations replaced by sliding window attention without performance loss.

Identified a long-tail distribution in context dependency, with decreasing need for full attention as local window size increases.

Decoupling local and global processing reveals full attention is largely redundant for many tasks.

Abstract

When reading books, humans focus primarily on the current page, flipping back to recap prior context only when necessary. Similarly, we demonstrate that Large Language Models (LLMs) can learn to dynamically determine when to attend to global context. We propose All-or-Here Attention (AHA), which utilizes a binary router per attention head to dynamically toggle between full attention and local sliding window attention for each token. Our results indicate that with a window size of 256 tokens, up to 93\% of the original full attention operations can be replaced by sliding window attention without performance loss. Furthermore, by evaluating AHA across various window sizes, we identify a long-tail distribution in context dependency, where the necessity for full attention decays rapidly as the local window expands. By decoupling local processing from global access, AHA reveals that full attention is largely redundant, and that efficient inference requires only on-demand access to the global context.