BestServe: Serving Strategies with Optimal Goodput in Collocation and Disaggregation Architectures

TL;DR

BestServe is a framework that efficiently predicts optimal serving strategies for large language models, reducing planning time and cost while maintaining high accuracy across different architectures.

Contribution

It introduces a novel, lightweight framework that estimates goodput for LLM serving strategies using an inference simulator based on an adapted roofline model.

Findings

Predicts serving strategy goodput within 20% error margin

Determines optimal strategies in minutes on a standard CPU

Supports both collocated and disaggregated architectures

Abstract

Serving large language models (LLMs) to millions of users requires efficient resource allocation and parallelism strategies. It is a labor intensive trial-and-error process to find such a strategy. We present BestServe, a novel framework for ranking serving strategies by estimating goodput under various operating scenarios. Supporting both collocated and disaggregated architectures, BestServe leverages an inference simulator built on an adapted roofline model and CPU-GPU dispatch dynamics. Our framework determines the optimal strategy in minutes on a single standard CPU, eliminating the need for costly benchmarking, while achieving predictions within a $20\%$ error margin. It appeals to be practical for rapid deployment planning because of its lightweight design and strong extensibility.