ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production

TL;DR

This paper provides an in-depth characterization of real-world LLM serving workloads and introduces ServeGen, a framework for generating realistic workloads to improve benchmarking and resource provisioning.

Contribution

It offers the first comprehensive analysis of diverse LLM serving workloads and proposes ServeGen for realistic workload generation based on detailed insights.

Findings

ServeGen reduces under-provisioning by 50% in production scenarios.

Workload characterization covers language, multimodal, and reasoning models.

Realistic workload generation improves benchmarking accuracy.

Abstract

With the widespread adoption of Large Language Models (LLMs), serving LLM inference requests has become an increasingly important task, attracting active research advancements. Practical workloads play an essential role in this process: they are critical for motivating and benchmarking serving techniques and systems. However, the existing understanding of real-world LLM serving workloads is limited due to the lack of a comprehensive workload characterization. Prior analyses remain insufficient in scale and scope, thus failing to fully capture intricate workload characteristics. In this paper, we fill the gap with an in-depth characterization of LLM serving workloads collected from our worldwide cloud inference serving service, covering not only language models but also emerging multimodal and reasoning models, and unveiling important new findings in each case. Moreover, based on our findings, we propose ServeGen, a principled framework for generating realistic LLM serving workloads by composing them on a per-client basis. A practical use case in production validates that ServeGen avoids 50% under-provisioning compared to naive workload generation, demonstrating ServeGen's advantage in performance benchmarking. ServeGen is available at https://github.com/alibaba/ServeGen.