Preble: Efficient Distributed Prompt Scheduling for LLM Serving

TL;DR

Preble is a novel distributed prompt scheduling platform for large language models that significantly reduces latency by optimizing KV state reuse and load balancing across multiple GPUs.

Contribution

It introduces the first distributed prompt scheduling system that co-optimizes KV reuse and computation load balancing for scalable LLM serving.

Findings

Preble achieves 1.5X to 14.5X latency reduction over state-of-the-art systems.

Preble reduces p99 latency by 2X to 10X.

Evaluation on real workloads demonstrates substantial performance improvements.

Abstract

Prompts to large language models (LLMs) have evolved beyond simple user questions. For LLMs to solve complex problems, today's practices are to include domain-specific instructions, illustration of tool usages, and/or long context such as textbook chapters in prompts. As such, many parts of prompts are repetitive across requests. Recent works propose to cache and reuse KV state of prompts. However, they are all confined to a single-GPU optimization, while production LLM serving systems are distributed by nature. This paper proposes Preble, the first distributed LLM serving platform that targets and optimizes for prompt sharing. We designed a distributed scheduling system that co-optimizes KV state reuse and computation load-balancing with a new scheduling algorithm and a hierarchical scheduling mechanism. Our evaluation of Preble with real workloads and request arrival patterns on two open-source LLMs shows that Preble outperforms the SOTA serving systems by 1.5X to 14.5X on average latency and 2X to 10X on p99 latency.