Quasar: Quantized Self-Speculative Acceleration for Rapid Inference via Memory-Efficient Verification

TL;DR

Quasar is a training-free, quantization-based framework that accelerates LLM inference by reducing memory bandwidth during verification, maintaining accuracy and improving throughput by 1.28 times.

Contribution

It introduces a novel quantization approach for verification in speculative decoding, effectively overcoming the memory bandwidth bottleneck without retraining.

Findings

Quantization preserves logit distribution fidelity.

Achieves 1.28x throughput improvement on large models.

Maintains verification accuracy comparable to full-precision methods.

Abstract

Speculative Decoding (SD) has emerged as a premier technique for accelerating Large Language Model (LLM) inference by decoupling token generation into rapid drafting and parallel verification. While recent advancements in self-speculation and lookahead decoding have successfully minimized drafting overhead, they have shifted the primary performance bottleneck to the verification phase. Since verification requires a full forward pass of the target model, it remains strictly memory-bandwidth bound, fundamentally limiting the maximum achievable speedup.In this paper, we introduce \textbf{Quasar} (\textbf{Qua}ntized \textbf{S}elf-speculative \textbf{A}cceleration for \textbf{R}apid Inference), a novel, training-free framework designed to overcome this "memory wall" by employing low-bit quantization specifically for the verification stage. Our empirical analysis reveals that while aggressive structural pruning significantly degrades verification accuracy, quantization-based verification preserves the logit distribution with high fidelity while effectively halving memory traffic. Extensive experiments on state-of-the-art models (e.g., OpenPangu and Qwen3) demonstrate that Quasar maintains a speculative acceptance length comparable to full-precision methods while achieving a $1.28\times$ improvement in end-to-end throughput. Being orthogonal to existing drafting strategies, Quasar offers a generic and efficient pathway to accelerate the verification leg of speculative execution. Code is available at https://github.com/Tom-HG/Quasar.