DSSD: Efficient Edge-Device LLM Deployment and Collaborative Inference via Distributed Split Speculative Decoding

TL;DR

DSSD introduces a distributed approach to collaborative LLM inference that reduces communication overhead and latency on edge devices while maintaining accuracy, enabling more efficient deployment of large language models in resource-constrained environments.

Contribution

It proposes a novel distributed split speculative decoding architecture that partitions verification between device and edge, significantly reducing communication costs and latency.

Findings

Reduces uplink transmission by replacing multiple vocabulary distributions with a single downlink.

Maintains inference accuracy comparable to existing methods.

Outperforms current solutions in efficiency and communication overhead.

Abstract

Large language models (LLMs) have transformed natural language processing but face critical deployment challenges in device-edge systems due to resource limitations and communication overhead. To address these issues, collaborative frameworks have emerged that combine small language models (SLMs) on devices with LLMs at the edge, using speculative decoding (SD) to improve efficiency. However, existing solutions often trade inference accuracy for latency or suffer from high uplink transmission costs when verifying candidate tokens. In this paper, we propose Distributed Split Speculative Decoding (DSSD), a novel architecture that not only preserves the SLM-LLM split but also partitions the verification phase between the device and edge. In this way, DSSD replaces the uplink transmission of multiple vocabulary distributions with a single downlink transmission, significantly reducing communication latency while maintaining inference quality. Experiments show that our solution outperforms current methods, and codes are at: https://github.com/JasonNing96/DSSD-Efficient-Edge-Computing