HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computing

TL;DR

HybridGen is a novel CPU-GPU hybrid attention framework that improves long-context LLM inference efficiency by addressing memory and bandwidth challenges with innovative parallelism, scheduling, and cache mapping techniques.

Contribution

It introduces a hybrid attention approach leveraging tiered memory systems, enabling better utilization of CPU and GPU resources for long-context LLM inference.

Findings

HybridGen outperforms existing KV cache methods by 1.41x to 3.2x on average.

Experiments on three LLM models and multiple GPU platforms demonstrate significant efficiency gains.

HybridGen maintains high accuracy while improving inference speed.

Abstract

As modern LLMs support thousands to millions of tokens, KV caches grow to hundreds of gigabytes, stressing memory capacity and bandwidth. Existing solutions, such as KV cache pruning and offloading, alleviate these but underutilize hardware by relying solely on either GPU or CPU for attention computing, and considering yet limited CPU local memory for KV cache storage. We propose HybridGen, an efficient hybrid attention framework for long-context LLM inference. HybridGen enables CPU-GPU collaborative attention on systems with expanded tiered memory (e.g., CXL memory), addressing three key challenges: (1) multi-dimensional attention dependencies, (2) intensifying CPU-GPU load imbalance with longer sequences, and (3) NUMA penalty of tiered memories. HybridGen tackles these by introducing attention logit parallelism, a feedback-driven scheduler, and semantic-aware KV cache mapping. Experiments with three LLM models with eleven different sizes on three GPU platforms with a CXL-expanded memory show that HybridGen outperforms six state-of-the-art KV cache management methods by 1.41x--3.2x on average while maintaining superior accuracy.