EPIC: Efficient Position-Independent Caching for Serving Large Language Models

TL;DR

EPIC introduces a novel position-independent caching system for large language models, enabling more efficient reuse of intermediate representations across requests with different prefixes, significantly improving serving speed and throughput.

Contribution

The paper presents EPIC, a new serving system with the LegoLink algorithm that enhances context caching by enabling prefix-agnostic reuse of KV vectors, addressing limitations of prior exact prefix matching methods.

Findings

EPIC achieves up to 8x reduction in Time-To-First-Token.

EPIC attains 7x throughput gains over existing systems.

EPIC maintains accuracy with minimal or no loss.

Abstract

Large Language Models (LLMs) show great capabilities in a wide range of applications, but serving them efficiently becomes increasingly challenging as requests (prompts) become more complex. Context caching improves serving performance by reusing Key-Value (KV) vectors, the intermediate representations of tokens that are repeated across requests. However, existing context caching requires exact prefix matches across requests, limiting reuse cases in settings such as few-shot learning and retrieval-augmented generation, where immutable content (e.g., documents) remains unchanged across requests but is preceded by varying prefixes. Position-Independent Caching (PIC) addresses this issue by enabling modular reuse of the KV vectors regardless of prefixes. We formalize PIC and advance prior work by introducing EPIC, a serving system incorporating our new LegoLink algorithm, which mitigates the inappropriate "attention sink" effect at every document beginning, to maintain accuracy with minimal computation. Experiments show that EPIC achieves up to 8x improvements in Time-To-First-Token (TTFT) and 7x throughput gains over existing systems, with negligible or no accuracy loss.