A Universal Load Balancing Principle and Its Application to Large Language Model Serving

TL;DR

This paper introduces a universal load balancing principle for barrier-synchronized systems, significantly reducing energy waste and improving performance in large language model serving and similar parallel computing systems.

Contribution

It develops a provably effective load balancing method with theoretical guarantees, applicable to diverse barrier-synchronized systems with non-migratable state.

Findings

Energy savings exceeding 52% at fleet scale

28% reduction in energy consumption in experiments

Substantial throughput and latency improvements

Abstract

Over 40% of computational power in Large Language Model (LLM) serving systems can be systematically wasted - not from hardware limits, but from load imbalance in barrier-synchronized parallel processing. When progress is gated by the slowest worker at each step, heterogeneous and evolving workloads create persistent stragglers; faster workers idle while drawing power, producing nothing. In large language model inference alone, this translates to gigawatt-hours of wasted electricity daily. Here we develop a universal load-balancing principle for barrier-synchronized systems with non-migratable state. We prove worst-case theoretical guarantees: imbalance reduction grows with system scale, and the resulting energy savings can exceed 52% for modern hardware at fleet scale. Experiments corroborate the theory, demonstrating 28% energy reduction alongside substantial throughput and latency improvements. Formulated as an online integer optimization with provable guarantees, the principle extends beyond LLM serving to broad classes of barrier-synchronized parallel systems, establishing a theoretical foundation for sustainable high-performance computing.