InstCache: A Predictive Cache for LLM Serving

TL;DR

InstCache introduces a predictive caching system for LLM inference that leverages LLM capabilities to reorder instruction representations, significantly improving cache hit rates and reducing inference latency.

Contribution

The paper presents InstCache, a novel predictive caching mechanism that uses LLMs to enhance instruction-level cache efficiency in LLM serving systems.

Findings

Achieves 2.3x higher cache hit rate than traditional methods.

Reduces time per output token by up to 42% and 50%.

Demonstrates effectiveness on multiple datasets.

Abstract

The revolutionary capabilities of Large Language Models (LLMs) are attracting rapidly growing popularity and leading to soaring user requests to inference serving systems. Caching techniques, which leverage data reuse to reduce computation, offer opportunities to optimize the performance of LLM inference engines. On the one hand, the low-level key-value (KV) cache working at the token level is widely adopted, albeit it incurs significant overhead as request volume grows. On the other hand, instruction-level caching, which stores full instruction-response pairs, is expected to play an increasingly crucial role. However, the high variability in the content and length of instructions make it rare for identical instructions to recur within a short time window, presenting challenges for effective caching instruction-response pairs. To address this challenge, we propose InstCache, a predictive caching mechanism for LLM serving systems. Leveraging the capability of LLMs, we can effectively reorder the representation space of instruction texts and develop a sufficient level of spatial locality. Such spatial locality enables us to predict potential instructions located in a compact region in the space, resulting in an effective caching system at runtime. Experimental results demonstrate that InstCache achieves a 2.3x higher hit rate compared to the upper bound of traditional caching mechanisms on WildChat dataset and reduces the time per output token of vLLM by up to 42.0% and 50.0% on LMSys and Moss datasets, respectively.