MultiPath Memory Access: Breaking Host-GPU Bandwidth Bottlenecks in LLM Services

TL;DR

MMA is a novel software system that enables efficient multipath host-GPU data transfer in multi-GPU servers, significantly improving bandwidth and reducing latency for large language model serving.

Contribution

It introduces the first software-defined multipath host-GPU data transfer system that leverages intra-server links without hardware or driver modifications.

Findings

Achieves 245 GB/s peak bandwidth, 4.62x faster than native CUDA copies.

Reduces KV cache fetch TTFT by up to 2.38x.

Lowers model wake-up latency by up to 2.48x.

Abstract

Host-GPU data movement has become a latency-critical bottleneck in LLM serving, surfacing in common paths such as model-weight movement and KV cache offload/fetch. Today, each host-GPU copy is effectively confined to the PCIe path of the target GPU, even though modern multi-GPU servers contain additional PCIe links on peer GPUs and high bandwidth GPU interconnects. This leaves substantial intra-server I/O capacity unused. To address this issue, we present Multipath Memory Access (MMA), a software-defined multipath memory access system for host--GPU data transfer. To the best of our knowledge, MMA is the first software-defined system to enable efficient multipath host--GPU data transfer within a single multi-GPU server. MMA expands a single host--GPU copy across available direct and relay paths without hardware, driver, or application changes. It preserves CUDA stream semantics with a dependency-preserving Dummy Task, coordinates distributed micro-transfer completion through a lightweight synchronization mechanism, and uses queue backpressure to route traffic without explicit link-state feedback. On an 8-GPU NVIDIA H20 server, MMA achieves 245 GB/s peak host-to-GPU bandwidth, a 4.62x improvement over native CUDA copies, and reduces TTFT for KV cache fetching by 1.14-2.38x and model wake-up/switching latency by 1.12-2.48x.