ProxyKV: Cross-Model Proxy Pruning for Efficient Long-Context LLM Inference

TL;DR

ProxyKV introduces a cross-model proxy pruning framework that enhances long-context LLM inference efficiency by asynchronously offloading importance scoring to a lightweight proxy, achieving high accuracy and significant speedups.

Contribution

The paper presents ProxyKV, a novel cross-model proxy pruning method with a hybrid architecture and loss, improving long-context inference speed and accuracy across multiple LLM families.

Findings

Matches KVZip accuracy while significantly speeding up KV prefetching.

Achieves up to 3.21x speedup on Llama-3.1-8B with minimal accuracy loss.

Maintains speedup at context lengths up to 170k tokens on Qwen-2.5-7B.

Abstract

Efficient long-context inference in Large Language Models (LLMs) is severely constrained by the Key-Value (KV) cache memory wall, yet existing pruning methods force a choice between low-latency heuristics that sacrifice precision and high-precision reconstruction methods that incur prohibitive prefilling overhead. To bridge this scoring-cost--accuracy gap, we propose ProxyKV, a cross-model proxy pruning framework that offloads importance scoring to a lightweight intra-family Small-Model Proxy executed asynchronously to the Large-Model Target. To bridge the architectural gap between heterogeneous models, we design the HybridAxialMapper, which disentangles temporal feature extraction from cross-head alignment, together with a Multi-Granularity Hybrid Loss that shifts the learning objective from rigid regression to relative ranking consistency. Across the Llama-3.1, Qwen-2.5, and Qwen-3 families spanning targets from 7B up to 32B parameters on LongBench, SCBench, and RULER, ProxyKV matches KVZip on aggregate (recovering $\sim$$98.7\%$ of its mean accuracy) while delivering up to a $3.21\times$ prefilling speedup on Llama-3.1-8B (dual-GPU; $\sim$$1.5\times$ shared single-GPU) and sustaining the speedup at contexts up to 170k tokens on Qwen-2.5-7B.