Fast Large Language Model Collaborative Decoding via Speculation

TL;DR

The paper introduces CoS, a speculative decoding framework that accelerates large language model collaborative decoding by using proposal and verification models in an alternating, parallel manner, achieving significant speedups without quality loss.

Contribution

It proposes a novel speculative decoding approach for collaborative LLM decoding, improving speed while maintaining output quality, with theoretical guarantees and extensive empirical validation.

Findings

CoS achieves 1.11x-2.23x speedup over standard methods.

Theoretical proof that CoS is never slower than standard decoding.

Maintains comparable generation quality with increased efficiency.

Abstract

Large Language Model (LLM) collaborative decoding techniques improve output quality by combining the outputs of multiple models at each generation step, but they incur high computational costs. In this paper, we introduce Collaborative decoding via Speculation (CoS), a novel framework that accelerates collaborative decoding without compromising performance. Inspired by Speculative Decoding--where a small proposal model generates tokens sequentially, and a larger target model verifies them in parallel, our approach builds on two key insights: (1) the verification distribution can be the combined distribution of both the proposal and target models, and (2) alternating each model as the proposer and verifier can further enhance efficiency. We generalize this method to collaboration among n models and theoretically prove that CoS is never slower than standard collaborative decoding, typically achieving faster speed. Extensive experiments demonstrate CoS is 1.11x-2.23x faster than standard collaborative decoding without compromising generation quality. Our code is available at https://github.com/Kamichanw/CoS/.