Slow-Fast Inference: Training-Free Inference Acceleration via Within-Sentence Support Stability

TL;DR

This paper introduces Slow-Fast Inference, a training-free decoding method that accelerates long-context autoregressive decoding by leveraging within-sentence support stability, achieving significant speedups without sacrificing quality.

Contribution

The paper proposes a novel training-free decoding framework that decouples inference into fast and slow steps based on semantic boundaries, improving efficiency in long-context models.

Findings

Achieves 1.6x to 14.4x higher decoding throughput.

Maintains comparable quality to full-KV baselines.

Applicable directly to existing models without retraining.

Abstract

Long-context autoregressive decoding remains expensive because each decoding step must repeatedly process a growing history. We observe a consistent pattern during decoding: within a sentence, and more generally within a short semantically coherent span, the dominant attention support often remains largely stable. Motivated by this observation, we propose Slow-Fast Inference (SFI), a training-free decoding framework that decouples generation into frequent low-cost fast steps and occasional dense-attention slow steps. Fast steps reuse a compact sparse memory for efficient decoding. Slow steps are triggered near semantic boundaries. At slow steps, the model revisits the broader context and uses the Selector to refresh the selected memory for subsequent fast steps. Across the evaluated context lengths, SFI delivers approximately $1.6\times$--$14.4\times$ higher decoding throughput while generally maintaining quality on par with the full-KV baseline across long-context and long-CoT settings. Because SFI is training-free and applies directly to existing checkpoints, it offers a practical path to reducing inference cost for contemporary autoregressive reasoning models in long-context, long-horizon, and agentic workloads.