ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments

TL;DR

ThunderServe is a novel system that enhances large language model serving in cloud environments by optimizing deployment and adapting to dynamic conditions, significantly improving throughput and reducing latency.

Contribution

It introduces a new scheduling algorithm and lightweight re-scheduling mechanism tailored for heterogeneous cloud resources, improving performance and cost-efficiency.

Findings

Up to 2.1× increase in throughput

Up to 2.5× reduction in latency

Better cost-efficiency compared to state-of-the-art systems

Abstract

Recent developments in large language models (LLMs) have demonstrated their remarkable proficiency in a range of tasks. Compared to in-house homogeneous GPU clusters, deploying LLMs in cloud environments with diverse types of GPUs is crucial for addressing the GPU shortage problem and being more cost-effective. However, the diversity of network environments and various GPU types on the cloud bring difficulties to achieving high-performance serving. In this work, we propose ThunderServe, a high-performance and cost-efficient LLM serving system for heterogeneous cloud environments. We introduce a novel scheduling algorithm, which optimizes the deployment plan of LLM serving to accommodate the heterogeneous resource and network bandwidth conditions in cloud environments. Furthermore, we propose a lightweight re-scheduling mechanism, designed to adapt to fluctuating online conditions (e.g., node failures, workload shifts) without the need for costly restarts of ongoing services. Empirical results in both heterogeneous cloud and homogeneous in-house environments reveal that ThunderServe delivers up to a 2.1$\times$ and on average a $1.7\times$ increase in throughput and achieves up to a 2.5$\times$ and on average a $1.5\times$ reduction in latency deadlines compared with state-of-the-art systems given the same price budget, suggesting opting for cloud services provides a more cost-efficient solution.