ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference

TL;DR

ScoutAttention is a novel framework that improves large language model inference efficiency by offloading KV cache to CPU with layer-ahead pre-computation and collaborative attention, reducing GPU memory constraints.

Contribution

It introduces a layer-ahead CPU pre-computation algorithm and collaborative sparse attention to accelerate inference while maintaining accuracy.

Findings

Achieves 2.1x speedup over existing offloading methods.

Maintains inference accuracy within 2.4% of baseline.

Reduces CPU load with collaborative sparse attention.

Abstract

Large language models encounter critical GPU memory capacity constraints during long-context inference, where KV cache memory consumption severely limits decode batch sizes. While existing research has explored offloading KV cache to DRAM, these approaches either demand frequent GPU-CPU data transfers or impose extensive CPU computation requirements, resulting in poor GPU utilization as the system waits for I/O operations or CPU processing to complete. We propose ScoutAttention, a novel KV cache offloading framework that accelerates LLM inference through collaborative GPU-CPU attention computation. To prevent CPU computation from bottlenecking the system, ScoutAttention introduces GPU-CPU collaborative block-wise sparse attention that significantly reduces CPU load. Unlike conventional parallel computing approaches, our framework features a novel layer-ahead CPU pre-computation algorithm, enabling the CPU to initiate attention computation one layer in advance, complemented by asynchronous periodic recall mechanisms to maintain minimal CPU compute load. Experimental results demonstrate that ScoutAttention maintains accuracy within 2.4% of baseline while achieving 2.1x speedup compared to existing offloading methods.