WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving

TL;DR

WarmServe introduces a proactive GPU prewarming approach for multi-LLM serving, leveraging workload predictability to significantly reduce latency and improve throughput in shared GPU environments.

Contribution

The paper presents WarmServe, a novel system with algorithms for workload-aware prewarming, GPU memory management, and interference minimization, enabling efficient multi-LLM deployment.

Findings

Reduces tail TTFT by up to 50.8 times compared to autoscaling systems.

Supports up to 2.5 times higher request throughput than GPU-sharing systems.

Leverages workload periodicity for proactive model preloading.

Abstract

Deploying multiple models within shared GPU clusters is a key strategy to improve resource efficiency in large language model (LLM) serving. Existing multi-LLM serving systems improve GPU utilization at the cost of degraded inference performance, particularly time-to-first-token (TTFT). We attribute this degradation to the lack of awareness regarding future workload characteristics. In contrast, recent analyses have shown the strong periodicity and long-term predictability of real-world LLM serving workloads. In this paper, we propose one-for-many GPU prewarming, which proactively loads parameters from multiple models onto GPUs based on workload forecasts. These prewarmed weights enable the system to promptly instantiate serving instances upon encountering request bursts. We design and implement WarmServe, a multi-LLM serving system incorporating three key techniques: (1) a model placement algorithm that optimizes prewarming decisions to minimize cross-model prewarming interference, (2) a KV cache reservation strategy that repurposes idle KV cache space on running GPUs for prewarming new models, and (3) an efficient GPU memory switching mechanism for tensor management. Evaluation on real-world datasets shows that WarmServe reduces tail TTFT by up to 50.8$\times$ compared to the state-of-the-art autoscaling-based system, while supporting up to 2.5$\times$ higher request throughput than the GPU-sharing system.