Characterizing CPU-Induced Slowdowns in Multi-GPU LLM Inference

TL;DR

This paper investigates how CPU limitations cause performance bottlenecks in multi-GPU large language model inference, revealing that increasing CPU resources can significantly improve throughput and latency without extra GPUs.

Contribution

It systematically analyzes CPU-induced slowdowns in multi-GPU LLM inference and demonstrates that augmenting CPU cores enhances performance and stability at minimal cost.

Findings

CPU bottlenecks cause GPU underutilization and increased latency.

Adding CPU cores reduces time-to-first-token by up to 5.40x.

CPU provisioning is critical for optimal multi-GPU LLM inference performance.

Abstract

Large-scale machine learning workloads increasingly rely on multi-GPU systems, yet their performance is often limited by an overlooked component: the CPU. Through a detailed study of modern large language model (LLM) inference and serving workloads, we find that multi-GPU performance frequently degrades not because GPUs are saturated, but because CPUs fail to keep the GPUs busy. Under limited CPU allocations, systems exhibit symptoms such as delayed kernel launch, stalled communication, and increased tokenization latency, leading to severe GPU underutilization even when ample GPU resources are available. This work presents a systematic analysis of CPU-induced slowdowns in multi-GPU LLM inference. We show that these bottlenecks persist even in serving stacks that employ process-level separation and modern GPU-side optimizations such as CUDA Graphs. Since the marginal cost of additional CPU cores is small relative to GPU instance pricing, our evaluation indicates that increasing the number of CPU cores can substantially improve performance and stability at minimal additional cost. Under moderate serving load, we observe that CPU-starved configurations frequently time out, while providing adequate CPU resources restores responsiveness and reduces time-to-first-token (TTFT) latency by 1.36-5.40x across configurations, all without requiring additional GPUs. This work shows that CPU provisioning is a crucial factor in multi-GPU LLM inference configuration, helping prevent control-side bottlenecks.