KV Cache Recycling to Expand Usable Context Capacity in Low Parameter LLMs

TL;DR

This paper introduces a method called token recycling that reuses attention KV states in small LLMs to expand context capacity and accelerate inference without model modifications.

Contribution

It demonstrates a practical approach to reuse cached KV states for small LLMs, improving inference speed when prompts share prefixes.

Findings

Speedups observed with prefix overlap

No degradation in output semantics

Reproducible with no model modifications

Abstract

Whether attention key value (KV) states computed for one prompt for a small LLM can be reused to accelerate inference on a new similar prompt, giving an increase to the space to its context memory using an approach called token recycling. Using a standard Hugging Face setup with DialoGPT-medium (a 345M parameter GPT-2 style decoder trained on 147M Reddit exchanges, 2005 to 2017) as the testbed, we build a cache of past activations and get entries by sentence embeddings, then reuse cached past key values when the cached prompt is an exact prefix of the new input. We compare recycled vs. baseline runs on latency and output fidelity, and log reuse depth in tokens. Reproducibility requires no model modifications, cached KVs are serialized to the CPU, reloaded, and supplied to the generate function to continue decoding from the cached prefix. In tests, we observe consistent speedups when prefix overlap exists, with no material degradation in output semantics, and when overlap is absent, behavior matches baseline.