HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inference

TL;DR

HGCA introduces a hybrid CPU-GPU attention mechanism that enables scalable, high-quality long-context LLM inference by combining dense and sparse attention, optimizing resource utilization and maintaining accuracy.

Contribution

It presents a novel hybrid attention approach that efficiently combines dense GPU attention with sparse CPU attention, supporting longer sequences without retraining.

Findings

Achieves higher throughput and scalability on commodity hardware.

Supports longer context lengths and larger batch sizes.

Outperforms existing sparse attention methods in accuracy and performance.

Abstract

Scaling inference for large language models (LLMs) is increasingly constrained by limited GPU memory, especially due to growing key-value (KV) caches required for long-context generation. While existing approaches offload KV caches to CPU memory or apply sparse attention to reduce GPU load, they often underutilize CPU compute resources and compromise accuracy. We present HGCA, a hybrid CPU-GPU attention mechanism that enables scalable, high-throughput LLM inference with near-full attention quality. HGCA performs dense attention on recently generated KV entries retained in GPU memory and parallel sparse attention on selected, salient KV entries in CPU memory. The attention outputs are efficiently merged using log-sum-exp fusion, minimizing PCIe transfer overhead. HGCA also introduces a finegrained, per-head sparsification strategy optimized for CPU execution, preserving contextual relevance while reducing computation. Our implementation seamlessly integrates into existing LLM frameworks without requiring model retraining. Experiments across diverse models and workloads show that HGCA achieves superior scalability, supports longer sequences and larger batch sizes, and outperforms existing sparse attention baselines in both performance and accuracy -- all on commodity GPU hardware.