Chimera: A Lossless Decoding Method for Accelerating Large Language Models Inference by Fusing all Tokens

TL;DR

Chimera introduces a novel speculative sampling framework with a lightweight draft model that accelerates LLM inference by 2.7 times while maintaining accuracy, addressing the resource-intensive decoding bottleneck.

Contribution

The paper proposes Chimera, a new speculative sampling framework with a lightweight draft model that improves decoding speed without sacrificing accuracy.

Findings

Achieves 2.7x latency speedup on Vicuna and LLaMA-2 models

Effectively captures short-range dependencies and leverages original LLM representations

Demonstrates significant efficiency improvements in large language model decoding

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities across various tasks. However, their widespread application is hindered by the resource-intensive decoding process. To address this challenge, current approaches have incorporated additional decoding heads to enable parallel prediction of multiple subsequent tokens, thereby achieving inference acceleration. Nevertheless, the accuracy of these decoding heads falls short of the auto-regressive decoding approach. In light of these limitations, we propose Chimera, a novel framework specifically designed for speculative sampling. Within this framework, we introduce a lightweight draft model that effectively utilizes previously generated tokens to predict subsequent words. To ensure both accuracy and efficiency, we present two strategies within the lightweight draft model. Firstly, we focus on capturing short-range dependencies at the bottom layer. Secondly, we leverage the readily available representations from the original LLM.Through empirical evaluation on the Vicuna and LlaMA-2 series, Chimera demonstrates impressive results, achieving an average latency speedup ratio of 2.7x compared to the vanilla auto-regressive decoding approach. This highlights the potential of our proposed framework in significantly improving the efficiency of large language models during the decoding process.