PackInfer: Compute- and I/O-Efficient Attention for Batched LLM Inference

TL;DR

PackInfer is a kernel-level framework that improves large language model inference efficiency by balancing compute and I/O, reducing latency and increasing throughput in heterogeneous batched requests.

Contribution

It introduces a compute- and I/O-aware execution method that optimally packs requests and reorganizes data to enhance GPU utilization during batched LLM inference.

Findings

Reduces inference latency by 13.0-20.1%.

Improves throughput by 20% over FlashAttention.

Effectively balances GPU workload for heterogeneous request batches.

Abstract

Attention efficiency is critical to large language model (LLM) inference. While prior advances optimize attention execution for individual requests (e.g., FlashAttention), production LLM serving relies on batching requests with highly heterogeneous sequence lengths for high serving throughput. This mismatch induces severe computation and I/O imbalance, exacerbates stragglers, and underutilizes GPU resources. We present PackInfer, a kernel-level attention framework that enables compute- and I/O-aware execution for heterogeneous batched inference. PackInfer orchestrates batched requests into load-balanced execution groups, effectively saturating GPU utilization by packing multiple requests into unified kernel launches. By constructing attention kernels directly over packed query-key regions, PackInfer eliminates redundant computation and balances thread-block execution. It then incorporates I/O-aware grouping that co-locates shared-prefix requests and reorganizes KV caches into group-contiguous layouts, reducing memory fragmentation and redundant data movement as generation evolves. Evaluations on real-world workloads show that PackInfer reduces inference latency by 13.0-20.1%, and improves throughput by 20% compared to the state-of-the-art FlashAttention.