MARS: Unleashing the Power of Speculative Decoding via Margin-Aware Verification

TL;DR

This paper introduces Margin-Aware Speculative Verification, a training-free method that improves the efficiency of speculative decoding in large language models by adaptively relaxing verification based on local decisiveness.

Contribution

It proposes a novel, domain-agnostic verification strategy that enhances speculative decoding efficiency without retraining, applicable across various model scales.

Findings

Achieves significant inference speedups over state-of-the-art methods.

Maintains generation quality across diverse benchmarks.

Effective across models from 8B to 235B parameters.

Abstract

Speculative Decoding (SD) accelerates autoregressive large language model (LLM) inference by decoupling generation and verification. While recent methods improve draft quality by tightly coupling the drafter with the target model, the verification mechanism itself remains largely unchanged, relying on strict token-level rejection sampling. In practice, modern LLMs frequently operate in low-margin regimes where the target model exhibits weak preference among top candidates. In such cases, rejecting plausible runner-up tokens yields negligible information gain while incurring substantial rollback cost, leading to a fundamental inefficiency in verification. We propose Margin-Aware Speculative Verification, a training-free and domain-agnostic verification strategy that adapts to the target model's local decisiveness. Our method conditions verification on decision stability measured directly from the target logits and relaxes rejection only when strict verification provides minimal benefit. Importantly, the approach modifies only the verification rule and is fully compatible with existing target-coupled speculative decoding frameworks. Extensive experiments across model scales ranging from 8B to 235B demonstrate that our method delivers consistent and significant inference speedups over state-of-the-art baselines while preserving generation quality across diverse benchmarks. The code is available at https://github.com/5SSjw/MARS.