Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding

TL;DR

This paper introduces self-speculative decoding, a novel inference method for LLMs that accelerates generation by drafting and verifying tokens without extra training or memory, achieving nearly double the speed.

Contribution

The paper proposes a training-free, memory-efficient decoding scheme that significantly speeds up LLM inference through a two-stage drafting and verification process.

Findings

Achieves up to 1.99× speedup on LLaMA-2 models.

No additional training or memory required.

Maintains output quality identical to original LLMs.

Abstract

We present a novel inference scheme, self-speculative decoding, for accelerating Large Language Models (LLMs) without the need for an auxiliary model. This approach is characterized by a two-stage process: drafting and verification. The drafting stage generates draft tokens at a slightly lower quality but more quickly, which is achieved by selectively skipping certain intermediate layers during drafting. Subsequently, the verification stage employs the original LLM to validate those draft output tokens in one forward pass. This process ensures the final output remains identical to that produced by the unaltered LLM. Moreover, the proposed method requires no additional neural network training and no extra memory footprint, making it a plug-and-play and cost-effective solution for inference acceleration. Benchmarks with LLaMA-2 and its variants demonstrated a speedup up to 1.99$\times$.