LLMCache: Layer-Wise Caching Strategies for Accelerated Reuse in Transformer Inference

TL;DR

LLMCache introduces a layer-wise caching framework that reuses intermediate transformer activations based on semantic similarity, significantly speeding up inference in models like BERT and GPT-2 with minimal accuracy loss.

Contribution

It proposes a model-agnostic, layer-wise caching method with semantic matching and adaptive eviction, extending caching beyond token-level approaches for transformer inference.

Findings

Up to 3.1x inference speedup on BERT and GPT-2.

Less than 0.5% accuracy degradation.

Effective across encoder and decoder architectures.

Abstract

Transformer-based language models have achieved remarkable performance across a wide range of tasks, yet their high inference latency poses a significant challenge for real-timeand large-scale deployment. While existing caching mechanisms,such as token-level key-value caches, offer speedups in autore-gressive decoding, they are limited in scope and applicability. In this paper, we present LLMCache, a novel layer-wise caching framework that accelerates transformer inference by reusing intermediate activations based on semantic similarity of input sequences. Unlike prior work, LLMCache is model-agnostic,operates across both encoder and decoder architectures, and supports caching at arbitrary transformer layers. We introduce a lightweight fingerprinting mechanism for matching seman-tically similar inputs and propose adaptive eviction strategies to manage cache staleness. Experiments on BERT and GPT-2 across SQuAD, WikiText-103, and OpenBookQA show up to 3.1 X speedup in inference time with <0.5% accuracy degradation. Our results highlight LLMCache as a practical and general-purpose solution for optimizing transformer inference in real-world applications