ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge--Cloud Speculative LLM Serving

TL;DR

ConfigSpec is a profiling-based framework for optimizing distributed speculative LLM serving across diverse edge devices and configurations, balancing throughput, cost, and energy efficiency.

Contribution

It introduces a profiling method to navigate the complex configuration space for distributed speculative LLM inference, revealing conflicting optimal points for different objectives.

Findings

Goodput maximized by smallest, fastest draft model at device-dependent lengths.

Cost and energy efficiency favor different draft sizes due to a bonus-token effect.

No single configuration optimizes all objectives, highlighting the need for profiling-based selection.

Abstract

Speculative decoding enables collaborative Large Language Model (LLM) inference across cloud and edge by separating lightweight token drafting from heavyweight verification. While prior systems show performance and cost benefits, practical deployment requires navigating a large configuration space spanning draft model variants, quantisation levels, speculative lengths, and heterogeneous edge devices. This paper presents ConfigSpec, a configurationselection framework for distributed speculative LLM serving. ConfigSpec profiles edge devices and draft-target alignment, and models drafting throughput, acceptance rate, and power to evaluate goodput, verification cost efficiency, and energy efficiency across the joint configuration space. Our analysis across three edge platforms and two LLM families reveals structurally conflicting optima. Firstly, goodput is maximised by the smallest, fastest draft model at device-dependent speculative lengths (K*=2-10). Secondly, both cost and energy efficiency converge to K=2 due to a dominant bonus-token effect-with cost favouring the largest drafter for its high acceptance rate and energy favouring the smallest for its low power draw. These conflicts confirm that no single fixed configuration can simultaneously optimise all objectives, underscoring the need for profiling-based configuration selection in disaggregated edge-cloud LLM inference.