Ensuring Fair LLM Serving Amid Diverse Applications

TL;DR

This paper introduces FairServe, a system designed to ensure fair access to large language models in multi-tenant platforms by accounting for application-specific request characteristics and preventing abuse.

Contribution

The paper develops FairServe, a novel fairness system that incorporates application-aware throttling and weighted scheduling, addressing limitations of existing methods in multi-tenant LLM serving.

Findings

FairServe outperforms existing fairness methods in real-world tests.

Application-aware request management improves fairness across diverse applications.

System deployment is underway to benefit millions of users.

Abstract

In a multi-tenant large language model (LLM) serving platform hosting diverse applications, some users may submit an excessive number of requests, causing the service to become unavailable to other users and creating unfairness. Existing fairness approaches do not account for variations in token lengths across applications and multiple LLM calls, making them unsuitable for such platforms. To address the fairness challenge, this paper analyzes millions of requests from thousands of users on MS CoPilot, a real-world multi-tenant LLM platform hosted by Microsoft. Our analysis confirms the inadequacy of existing methods and guides the development of FairServe, a system that ensures fair LLM access across diverse applications. FairServe proposes application-characteristic aware request throttling coupled with a weighted service counter based scheduling technique to curb abusive behavior and ensure fairness. Our experimental results on real-world traces demonstrate FairServe's superior performance compared to the state-of-the-art method in ensuring fairness. We are actively working on deploying our system in production, expecting to benefit millions of customers world-wide.