S2D: Sorted Speculative Decoding For More Efficient Deployment of Nested Large Language Models

TL;DR

This paper introduces S2D, a sorted speculative decoding method that enhances the efficiency of deploying nested large language models by improving multi-target inference speed and reducing costs.

Contribution

The paper proposes a novel sorted speculative decoding approach for multi-target LLM deployment, outperforming existing methods in efficiency and cost reduction.

Findings

Outperforms baseline methods in multi-target settings

Effective on models like Vicuna 7B, 13B, and LLama Chat 70B

Reduces inference costs for nested LLM deployment

Abstract

Deployment of autoregressive large language models (LLMs) is costly, and as these models increase in size, the associated costs will become even more considerable. Consequently, different methods have been proposed to accelerate the token generation process and reduce costs. Speculative decoding (SD) is among the most promising approaches to speed up the LLM decoding process by verifying multiple tokens in parallel and using an auxiliary smaller draft model to generate the possible tokens. In SD, usually, one draft model is used to serve a specific target model; however, in practice, LLMs are diverse, and we might need to deal with many target models or more than one target model simultaneously. In this scenario, it is not clear which draft model should be used for which target model, and searching among different draft models or training customized draft models can further increase deployment costs. In this paper, we first introduce a novel multi-target scenario for the deployment of draft models for faster inference. Then, we present a novel, more efficient sorted speculative decoding mechanism that outperforms regular baselines in multi-target settings. We evaluated our method on Spec-Bench in different settings, including base models such as Vicuna 7B, 13B, and LLama Chat 70B. Our results suggest that our draft models perform better than baselines for multiple target models at the same time.