EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization

TL;DR

EasySpec introduces a layer-parallel speculative decoding method that significantly accelerates multi-GPU LLM inference by enabling simultaneous layer execution, achieving up to 4.17x speedup with minimal accuracy loss.

Contribution

It proposes a novel layer-parallel speculation strategy that breaks inter-layer dependencies, allowing multiple layers to run concurrently across GPUs without retraining.

Findings

Achieves up to 4.17x speedup in decoding

Drafting stage accelerated by up to 1.62x

Maintains original model distribution with minimal accuracy drop

Abstract

Speculative decoding is an effective and lossless method for Large Language Model (LLM) inference acceleration. It employs a smaller model to generate a draft token sequence, which is then verified by the original base model. In multi-GPU systems, inference latency can be further reduced through tensor parallelism (TP), while the optimal TP size of the draft model is typically smaller than that of the base model, leading to GPU idling during the drafting stage. We observe that such inefficiency stems from the sequential execution of layers, which is seemingly natural but actually unnecessary. Therefore, we propose EasySpec, a layer-parallel speculation strategy that optimizes the efficiency of multi-GPU utilization. EasySpec breaks the inter-layer data dependencies in the draft model, enabling multiple layers to run simultaneously across multiple devices as 'fuzzy' speculation. After each drafting-and-verification iteration, the draft model's key-value cache is calibrated in a single forward pass, preventing long-term fuzzy-error accumulation at minimal additional latency. EasySpec is a training-free and plug-in method. We evaluated EasySpec on several mainstream open-source LLMs, using smaller versions of models from the same series as drafters. The results demonstrate that EasySpec can achieve a peak speedup of 4.17x compared to vanilla decoding, while preserving the original distributions of the base LLMs. Specifically, the drafting stage can be accelerated by up to 1.62x with a maximum speculation accuracy drop of only 7%. The code is available at https://github.com/Yize-Wu/EasySpec.