{\lambda}Scale: Enabling Fast Scaling for Serverless Large Language Model Inference

TL;DR

{0}Scale introduces a serverless inference system that leverages RDMA networks and distributed execution to enable rapid scaling of large language models, significantly reducing latency and costs during workload spikes.

Contribution

It presents {0}Scale, a novel system combining high-speed network multicast and distributed inference to improve scaling efficiency for large models in serverless environments.

Findings

Achieves up to 5x tail-latency reduction

Reduces inference costs by 31.3%

Effectively handles bursty workloads

Abstract

Serverless computing has emerged as a compelling solution for cloud-based model inference. However, as modern large language models (LLMs) continue to grow in size, existing serverless platforms often face substantial model startup overhead. This poses a significant challenge in efficiently scaling model instances to accommodate dynamic, bursty workloads commonly observed in real-world inference services. In this paper, we introduce {\lambda}Scale, an efficient serverless inference system to achieve fast model scaling. The key idea behind {\lambda}Scale is to leverage high-speed RDMA networks between GPU nodes for fast model multicast, while enabling distributed inference execution during model transmission -- referred to as "execute-while-load". {\lambda}Scale proposes an efficient model scaling scheme, {\lambda}Pipe, which supports adaptive model multicast and dynamically constructs execution pipelines across receiving nodes for collaborative, distributed inference. Additionally, {\lambda}Scale supports efficient model management across GPU and host memory, allowing fast scaling for models across different storage tiers. Evaluation results show that {\lambda}Scale enables fast model scaling and effectively handles load spikes, achieving up to 5x tail-latency improvement and 31.3% cost reduction compared to state-of-the-art solutions on real-world LLM inference traces.