Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption

TL;DR

This paper empirically investigates multi-model LLM scheduling challenges, revealing how offloading and preemption impact throughput and resource management on heterogeneous hardware, guiding future system design.

Contribution

It provides detailed insights into the performance implications of offloading and preemption for diverse LLMs, informing the design of more efficient multi-model schedulers.

Findings

Offloading causes non-linear, model-dependent throughput degradation.

Preemption overhead is dominated by model state reload, varying across models and hardware.

Sequence length and interconnect bandwidth significantly affect data movement and execution efficiency.

Abstract

Modern deployments of Large Language Models (LLMs) increasingly require serving multiple models with diverse architectures, sizes, and specialization on shared, heterogeneous hardware. This setting introduces new challenges for resource allocation, dispatching, and scheduling, particularly under GPU memory constraints where partial CPU-GPU offloading and preemption become necessary. While existing systems primarily optimize throughput for a single model, comparatively little work addresses multi-model scheduling under these conditions. In this paper, we present an empirical study of how different LLMs behave across hardware platforms, focusing on the performance implications of layer offloading and preemption. We show that offloading leads to strongly non-linear and model-dependent degradation in decode throughput, with smaller models exhibiting sharper sensitivity to reduced GPU residency. We further demonstrate that preemption incurs substantial overhead, largely dominated by model state reload rather than key-value cache transfer, and that this cost varies significantly across models and hardware platforms. Additionally, we highlight the role of sequence length and interconnect bandwidth in amplifying data movement and execution inefficiencies. Based on these findings, we identify a set of key features that future schedulers must consider, including model-specific offloading sensitivity, workload characteristics, and the cost structure of preemption and data transfer. These insights provide guidance for the design of next-generation LLM serving systems capable of efficiently managing heterogeneous, multi-model workloads with hybrid CPU-GPU execution.