Less Is More: Fast and Accurate Reasoning with Cross-Head Unified Sparse Attention

TL;DR

LessIsMore introduces a training-free sparse attention method that enhances long-horizon reasoning efficiency by maintaining stable, globally shared token importance, leading to faster decoding without accuracy loss.

Contribution

It presents a novel, training-free sparse attention mechanism that enforces cross-head unified token selection and stable context preservation for improved reasoning performance.

Findings

Matches or improves accuracy with fewer attended tokens.

Achieves up to 1.6x decoding speedup and 1.72x faster sparse attention.

Demonstrates effectiveness across multiple models and benchmarks.

Abstract

Large reasoning models achieve strong performance through test-time scaling, but this incurs substantial computational overhead due to long decoding from short prompts. While sparse attention can reduce latency and memory usage, existing methods often degrade reasoning accuracy because selection errors accumulate over long generation horizons, or require costly retraining. We introduce LessIsMore, a training-free sparse attention mechanism for long-horizon reasoning. Our key insight is that token importance in reasoning is global and stable: critical tokens are largely shared across attention heads and remain stable over decoding steps. Guided by this structure, LessIsMore enforces cross-head unified token selection and preserves recent context via a stable recency window, yielding a globally consistent token set that can be reused across layers. Across multiple model families and challenging reasoning benchmarks, LessIsMore matches or improves accuracy while attending to substantially fewer tokens. With kernel-level optimizations, LessIsMore achieves up to $1.6\times$ end-to-end decoding speedup and up to $1.72\times$ faster sparse attention computation, with additional long-context results demonstrating the generality of our approach. Code is available at \href{https://github.com/DerrickYLJ/LessIsMore}{https://github.com/DerrickYLJ/LessIsMore}.