Accelerate Speculative Decoding with Sparse Computation in Verification

TL;DR

This paper introduces a sparse verification framework for speculative decoding in language models, reducing computational bottlenecks by sparsifying multiple model components and reusing computations, leading to improved efficiency without sacrificing accuracy.

Contribution

It systematically applies sparse methods to the verification stage of speculative decoding, jointly sparsifies attention, FFN, and MoE components, and introduces reuse strategies to enhance efficiency.

Findings

Achieves significant speedup in various NLP tasks.

Maintains stable acceptance length and accuracy.

Demonstrates favorable efficiency-accuracy trade-offs.

Abstract

Speculative decoding accelerates autoregressive language model inference by verifying multiple draft tokens in parallel. However, the verification stage often becomes the dominant computational bottleneck, especially for long-context inputs and mixture-of-experts (MoE) models. Existing sparsification methods are designed primarily for standard token-by-token autoregressive decoding to remove substantial computational redundancy in LLMs. This work systematically adopts different sparse methods on the verification stage of the speculative decoding and identifies structured redundancy across multiple dimensions. Based on these observations, we propose a sparse verification framework that jointly sparsifies attention, FFN, and MoE components during the verification stage to reduce the dominant computation cost. The framework further incorporates an inter-draft token and inter-layer retrieval reuse strategy to further reduce redundant computation without introducing additional training. Extensive experiments across summarization, question answering, and mathematical reasoning datasets demonstrate that the proposed methods achieve favorable efficiency-accuracy trade-offs, while maintaining stable acceptance length.