FREESH: Fair, Resource- and Energy-Efficient Scheduling for LLM Serving on Heterogeneous GPUs

TL;DR

FREESH is a scheduling system that optimizes resource and energy efficiency for large language model serving across heterogeneous, geographically distributed GPU clusters, reducing energy consumption and emissions while maintaining performance.

Contribution

It introduces a joint routing and scheduling framework that leverages spatiotemporal flexibility and GPU heterogeneity to minimize energy and carbon footprint in LLM serving.

Findings

Reduced energy consumption by 28.6% in experiments.

Lowered emissions by 45.45% during testing.

Improved fairness and SLO adherence in production workloads.

Abstract

The ever-increasing computation and energy demand for LLM and AI agents call for holistic and efficient optimization of LLM serving systems. In practice, heterogeneous GPU clusters can be deployed in a geographically distributed manner, while LLM load also observes diversity in terms of both query traffic and serving patterns. LLM queries running on advanced GPUs during a high-emission hour at one location can lead to significantly higher carbon footprints versus same queries running on mid-level GPUs at a low-emission time and location. By observing LLM serving requirements and leveraging spatiotemporal computation flexibility, we consider the joint routing and scheduling problem, and propose FREESH to cooperatively run a group of data centers while minimizing user-specified carbon or energy objectives. FREESH identifies the optimal configurations of balanced load serving by matching distinct GPU instance's power-throughput characteristics with predictable LLM query length and workloads. To ensure both latency and fairness requirements, FREESH identifies optimized parallelism and query routing schedules together with dynamic GPU frequency scaling for power saving, and Least-Laxity-First (LLF) serving strategy for query scheduling. During the 1-hour serving on production workloads, FREESH reduces energy by 28.6% and emissions by 45.45% together with improvements in SLO attainment and fairness.