Zipage: Maintain High Request Concurrency for LLM Reasoning through Compressed PagedAttention

TL;DR

This paper introduces Zipage, a high-concurrency LLM inference engine that uses Compressed PagedAttention to efficiently manage memory and maintain high request throughput during reasoning tasks.

Contribution

The paper presents Compressed PagedAttention and a scheduling strategy, enabling high concurrency and memory efficiency in LLM inference, which is practical for industrial applications.

Findings

Achieves 95% of full KV inference performance on large-scale tasks.

Provides over 2.1× speedup in inference throughput.

Supports prefix caching and asynchronous compression for efficiency.

Abstract

With reasoning becoming the generative paradigm for large language models (LLMs), the memory bottleneck caused by KV cache during the decoding phase has become a critical factor limiting high-concurrency service. Although existing KV cache eviction methods address the memory issue, most of them are impractical for industrial-grade applications. This paper introduces Compressed PagedAttention, a method that combines token-wise KV cache eviction with PagedAttention. We propose a comprehensive scheduling strategy and support prefix caching and asynchronous compression for Compressed PagedAttention. Based on this, we have developed a high-concurrency LLM inference engine, Zipage. On large-scale mathematical reasoning tasks, Zipage achieves around 95\% of the performance of Full KV inference engines while delivering over 2.1$\times$ speedup.