CopySpec: Accelerating LLMs with Speculative Copy-and-Paste Without Compromising Quality

TL;DR

CopySpec is a technique that accelerates large language model inference by identifying repeated sequences and copying them, significantly improving speed without sacrificing output quality across various datasets and models.

Contribution

We introduce CopySpec, a novel method that enhances inference speed of LLMs by efficiently copying repeated sequences, compatible with speculative decoding and tested on multiple datasets.

Findings

Achieves up to 3.08x speed-up on specific tasks

Seamlessly integrates with speculative decoding for 49% additional speed-up

Leverages larger contexts to further accelerate inference

Abstract

We introduce CopySpec, a simple yet effective technique to tackle the inefficiencies LLMs face when generating responses that closely resemble previous outputs or responses that can be verbatim extracted from context. CopySpec identifies repeated sequences in the model's chat history or context and speculates that the same tokens will follow, enabling seamless copying without compromising output quality and without requiring additional GPU memory. To evaluate the effectiveness of our approach, we conducted experiments using seven LLMs and five datasets: MT-Bench, CNN/DM, GSM8K, HumanEval, and our newly created dataset, MT-Redundant. MT-Redundant, introduced in this paper, transforms the second turn of MT-Bench into a request for variations of the first turn's answer, simulating real-world scenarios where users request modifications to prior responses. Our results demonstrate significant speed-ups: up to 2.35x on CNN/DM, 3.08x on the second turn of select MT-Redundant categories, and 2.66x on the third turn of GSM8K's self-correction tasks. Importantly, we show that CopySpec integrates seamlessly with speculative decoding, yielding an average 49% additional speed-up over speculative decoding for the second turn of MT-Redundant across all eight categories. While LLMs, even with speculative decoding, suffer from slower inference as context size grows, CopySpec leverages larger contexts to accelerate inference, making it a faster complementary solution. Our code and dataset are publicly available at https://github.com/RazvanDu/CopySpec.