PipeSD: An Efficient Cloud-Edge Collaborative Pipeline Inference Framework with Speculative Decoding

TL;DR

PipeSD is a novel cloud-edge pipeline inference framework that leverages speculative decoding and optimized scheduling to significantly accelerate large language model inference while reducing energy consumption.

Contribution

It introduces a pipeline scheduling mechanism and a flexible verification triggering method, enhancing efficiency and robustness in cloud-edge collaborative inference.

Findings

Achieves 1.16x-2.16x speedup over baselines.

Reduces energy consumption by 14.3%-25.3%.

Outperforms existing frameworks on real-world testbeds.

Abstract

Speculative decoding can significantly accelerate LLM inference, especially given that its cloud-edge collaborative deployment offers cloud workload offloading, offline robustness, and privacy enhancement. However, existing collaborative inference frameworks with speculative decoding are constrained by (i) sequential token generation and communication with low resource utilization, and (ii) inflexible cloud non-autoregressive verification (NAV) triggering that induces premature verification or costly rollbacks. In this paper, we propose PipeSD, an efficient cloud-edge collaborative pipeline inference framework with speculative decoding. PipeSD overlaps token generation and communication by a token-batch pipeline scheduling mechanism optimized by dynamic programming, and improves verification flexibility through a dual-threshold NAV triggering mechanism with a lightweight Bayesian optimization autotuner. We implement PipeSD using llama-cpp-python, PyTorch, and FastAPI, and evaluate it on a real-world cloud-edge testbed with two draft-target model pairs across four scenarios. Results show that PipeSD consistently outperforms state-of-the-art baselines, achieving 1.16x-2.16x speedup and reducing energy consumption by 14.3%-25.3%.