CoVSpec: Efficient Device-Edge Co-Inference for Vision-Language Models via Speculative Decoding

TL;DR

CoVSpec is a novel framework that enhances device-edge collaborative inference for vision-language models by reducing visual tokens, adaptively adjusting decoding, and decoupling verification to improve efficiency and reduce communication costs.

Contribution

It introduces a training-free visual token reduction, an adaptive drafting strategy, and a parallel branching mechanism for efficient device-edge VLM inference.

Findings

Achieves up to 2.21x higher throughput than target-only inference.

Reduces communication overhead by over 96% compared to baselines.

Maintains task accuracy while significantly improving efficiency.

Abstract

Vision-language models (VLMs) have demonstrated strong capabilities in multimodal perception and reasoning. However, deploying large VLMs on mobile devices remains challenging due to their substantial computational and memory demands. A practical alternative is device-edge co-inference, where a lightweight draft VLM on the mobile device collaborates with a larger target VLM on the edge server via speculative decoding. Nevertheless, directly extending speculative decoding to VLMs suffers from severe inefficiency due to excessive visual-token computation and high communication overhead. To address these challenges, we propose CoVSpec, an efficient collaborative speculative decoding framework for VLM inference. Specifically, we first develop a training-free visual token reduction framework that prunes redundant visual tokens on the mobile device by jointly considering query relevance, token activity, and low-rank dependency. Moreover, we design an adaptive drafting strategy that dynamically adjusts both the verification frequency and the draft length. In addition, we introduce a parallel branching mechanism with decoupled verification-correction to improve draft-side utilization during target-side verification and reduce correction-related transmission overhead. Experiments on multiple benchmarks show that CoVSpec achieves up to 2.21x higher throughput than target-only inference and reduces communication overhead by more than 96% compared with baselines, without compromising task accuracy.