KV Cache Offloading for Context-Intensive Tasks

TL;DR

This paper investigates KV-cache offloading for long-context tasks in large language models, revealing performance issues and proposing a simple strategy to improve accuracy in context-intensive scenarios.

Contribution

It introduces the Text2JSON benchmark for evaluating KV offloading on context-heavy tasks and proposes a straightforward method that enhances accuracy across multiple models.

Findings

Significant performance degradation observed with KV offloading on Text2JSON and other tasks.

Identified low-rank projection and unreliable landmarks as causes of poor accuracy.

Proposed a simple alternative strategy that improves accuracy across models.

Abstract

With the growing demand for long-context LLMs across a wide range of applications, the key-value (KV) cache has become a critical bottleneck for both latency and memory usage. Recently, KV-cache offloading has emerged as a promising approach to reduce memory footprint and inference latency while preserving accuracy. Prior evaluations have largely focused on tasks that do not require extracting large amounts of information from the context. In this work, we study KV-cache offloading on context-intensive tasks: problems where the solution requires looking up a lot of information from the input prompt. We create and release the Text2JSON benchmark, a highly context-intensive task that requires extracting structured knowledge from raw text. We evaluate modern KV offloading on Text2JSON and other context-intensive tasks and find significant performance degradation on both Llama 3 and Qwen 3 models. Our analysis identifies two key reasons for poor accuracy: low-rank projection of keys and unreliable landmarks, and proposes a simpler alternative strategy that significantly improves accuracy across multiple LLM families and benchmarks. These findings highlight the need for a comprehensive and rigorous evaluation of long-context compression techniques.