Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristics

TL;DR

This paper emphasizes the need for mathematical optimization and algorithmic foundations in LLM inference serving, moving beyond heuristics to achieve reliable, provable performance guarantees.

Contribution

It highlights the limitations of current heuristic-based methods and advocates for developing principled, model-based algorithms with theoretical guarantees for LLM serving.

Findings

Current serving systems rely on classical distributed computing heuristics.

Emerging research shows principled methods can match heuristic performance.

Mathematical models can enable algorithms with provable guarantees.

Abstract

This position paper argues that LLM inference serving has outgrown generic heuristics and now demands mathematical optimization and algorithmic foundations. Despite rapid advances in serving systems such as vLLM and SGLang, their algorithmic cores remain largely unchanged from classical distributed computing: request routing uses join-shortest-queue or round-robin, scheduling defaults to FIFO, and KV cache eviction follows LRU. These general-purpose policies ignore the distinctive structure of LLM inference--dynamically growing KV cache memory, prefill-decode phase asymmetry, unknown output lengths, and continuous batching constraints. We contend that the field must develop mathematical models capturing these characteristics, enabling the design of algorithms with provable performance guarantees across diverse workloads, rather than heuristics that may succeed in some scenarios but fail unpredictably in others. Emerging work at the intersection of operations research and ML systems demonstrates that principled methods can match or exceed heuristic performance while providing theoretical guarantees. We call on the community to recognize algorithmic design for LLM serving as a research frontier.