Cacheback: Speculative Decoding With Nothing But Cache

TL;DR

Cacheback Decoding is a simple, training-free speculative decoding method that uses LRU cache tables of token n-grams to accelerate LLM inference by exploiting language locality, achieving state-of-the-art results.

Contribution

It introduces a minimalist, model-agnostic decoding approach that leverages only cache tables, enabling fast inference without additional training or model modifications.

Findings

Achieves state-of-the-art performance among comparable methods.

Easily integrates into existing systems due to its simplicity.

Shows potential for rapid adaptation to new domains.

Abstract

We present Cacheback Decoding, a training-free and model-agnostic speculative decoding method that exploits the locality in language to accelerate Large Language Model (LLM) inference. Cacheback leverages only Least Recently Used (LRU) cache tables of token n-grams to generate draft sequences. Cacheback achieves state-of-the-art performance among comparable methods despite its minimalist design, and its simplicity allows easy integration into existing systems. Cacheback also shows potential for fast adaptation to new domains.