A Pipelined Collaborative Speculative Decoding Framework for Efficient Edge-Cloud LLM Inference

TL;DR

This paper introduces PicoSpec, a novel edge-cloud collaborative inference framework for LLMs that improves efficiency through asynchronous pipelining and sparse compression, achieving significant speedups.

Contribution

The paper presents a training-free speculative decoding framework with an asynchronous pipeline and sparse compression, addressing mutual waiting and communication latency in edge-cloud LLM inference.

Findings

Achieves up to 2.9x speedup over baseline methods.

Introduces a one-time compressed vocabulary transmission for rejection sampling.

Effectively balances edge and cloud computation for LLM inference.

Abstract

Recent advancements and widespread adoption of Large Language Models (LLMs) in both industry and academia have catalyzed significant demand for LLM serving. However, traditional cloud services incur high costs, while on-device inference alone faces challenges due to limited resources. Edge-cloud collaboration emerges as a key research direction to combine the strengths of both paradigms, yet efficiently utilizing limited network bandwidth while fully leveraging and balancing the computational capabilities of edge devices and the cloud remains an open problem. To address these challenges, we propose Pipelined Collaborative Speculative Decoding Framework (PicoSpec), a novel, general-purpose, and training-free speculative decoding framework for LLM edge-cloud collaborative inference. We design an asynchronous pipeline that resolves the mutual waiting problem inherent in vanilla speculative decoding within edge collaboration scenarios, which concurrently executes a Small Language Model (SLM) on the edge device and a LLM in the cloud. Meanwhile, to mitigate the significant communication latency caused by transmitting vocabulary distributions, we introduce separate rejection sampling with sparse compression, which completes the rejection sampling with only a one-time cost of transmitting the compressed vocabulary. Experimental results demonstrate that our solution outperforms baseline and existing methods, achieving up to 2.9 speedup.