APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs

TL;DR

APEX introduces a dynamic, profiling-informed scheduling strategy that enhances CPU-GPU parallelism for large language model inference, significantly improving throughput while maintaining latency on constrained GPU hardware.

Contribution

It presents APEX, a novel scheduler that predicts execution times to optimize CPU-GPU task overlap during hybrid LLM inference, outperforming existing static and heuristic methods.

Findings

APEX improves throughput by up to 96% on T4 GPUs.

APEX achieves up to 89% higher throughput on A10 GPUs.

APEX maintains latency while significantly increasing efficiency.

Abstract

Deploying large language models (LLMs) for online inference is often constrained by limited GPU memory, particularly due to the growing KV cache during auto-regressive decoding. Hybrid GPU-CPU execution has emerged as a promising solution by offloading KV cache management and parts of attention computation to the CPU. However, a key bottleneck remains: existing schedulers fail to effectively overlap CPU-offloaded tasks with GPU execution during the latency-critical, bandwidth-bound decode phase. This particularly penalizes real-time, decode-heavy applications (e.g., chat, Chain-of-Thought reasoning) which are currently underserved by existing systems, especially under memory pressure typical of edge or low-cost deployments. We present APEX, a novel, profiling-informed scheduling strategy that maximizes CPU-GPU parallelism during hybrid LLM inference. Unlike systems relying on static rules or purely heuristic approaches, APEX dynamically dispatches compute across heterogeneous resources by predicting execution times of CPU and GPU subtasks to maximize overlap while avoiding scheduling overheads. We evaluate APEX on diverse workloads and GPU architectures (NVIDIA T4, A10), using LLaMa-2-7B and LLaMa-3.1-8B models. Compared to GPU-only schedulers like vLLM, APEX improves throughput by 84% - 96% on T4 and 11% - 89% on A10 GPUs, while preserving latency. Against the best existing hybrid schedulers, it delivers up to 72% (T4) and 37% (A10) higher throughput in long-output settings. APEX significantly advances hybrid LLM inference efficiency on such memory-constrained hardware and provides a blueprint for scheduling in heterogeneous AI systems, filling a critical gap for efficient real-time LLM applications.