PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding

TL;DR

PipeSpec introduces a hierarchical pipeline framework for speculative decoding in large language models, enabling asynchronous execution and significantly improving inference speed across multiple tasks and models.

Contribution

It generalizes speculative decoding to a hierarchical pipeline, reducing stage dependencies and increasing hardware utilization for faster LLM inference.

Findings

Achieves up to 2.54× speedup over existing methods.

Demonstrates scalability with model depth in multi-device systems.

Validates effectiveness on text summarization and code generation tasks.

Abstract

Speculative decoding accelerates large language model inference by using smaller draft models to generate candidate tokens for parallel verification. However, current approaches are limited by sequential stage dependencies that prevent full hardware utilization. We present PipeSpec, a framework that generalizes speculative decoding to $k$ models arranged in a hierarchical pipeline, enabling asynchronous execution with lightweight coordination for prediction verification and rollback. Our analytical model characterizes token generation rates across pipeline stages and proves guaranteed throughput improvements over traditional decoding for any non-zero acceptance rate. We further derive closed-form expressions for steady-state verification probabilities that explain the empirical benefits of pipeline depth. Experimental results show that PipeSpec achieves up to 2.54$\times$ speedup while outperforming state-of-the-art methods. We validate PipeSpec across text summarization and code generation tasks using LLaMA 2 and 3 models, demonstrating that pipeline efficiency increases with model depth, providing a scalable approach to accelerating LLM inference on multi-device systems.