Speculative Decoding in Decentralized LLM Inference: Turning Communication Latency into Computation Throughput

TL;DR

This paper introduces Decentralized Speculative Decoding (DSD), a framework that leverages speculative decoding in distributed LLM inference to turn communication latency into increased throughput, achieving significant speedups without retraining.

Contribution

The paper proposes DSD, a novel decentralized speculative decoding framework with adaptive verification, reducing communication costs and improving inference speed in distributed LLM systems.

Findings

Achieves up to 2.56x speedup on HumanEval

Achieves up to 2.59x speedup on GSM8K

Reduces cross-node communication cost significantly

Abstract

Speculative decoding accelerates large language model (LLM) inference by using a lightweight draft model to propose tokens that are later verified by a stronger target model. While effective in centralized systems, its behavior in decentralized settings, where network latency often dominates compute, remains under-characterized. We present Decentralized Speculative Decoding (DSD), a plug-and-play framework for decentralized inference that turns communication delay into useful computation by verifying multiple candidate tokens in parallel across distributed nodes. We further introduce an adaptive speculative verification strategy that adjusts acceptance thresholds by token-level semantic importance, delivering an additional 15% to 20% end-to-end speedup without retraining. In theory, DSD reduces cross-node communication cost by approximately (N-1)t1(k-1)/k, where t1 is per-link latency and k is the average number of tokens accepted per round. In practice, DSD achieves up to 2.56x speedup on HumanEval and 2.59x on GSM8K, surpassing the Eagle3 baseline while preserving accuracy. These results show that adapting speculative decoding for decentralized execution provides a system-level optimization that converts network stalls into throughput, enabling faster distributed LLM inference with no model retraining or architectural changes.