GAR: Carbon-Aware Routing for LLM Inference via Constrained Optimization

TL;DR

This paper introduces Green-Aware Routing (GAR), a multi-objective optimization framework for LLM inference that reduces CO2 emissions while maintaining accuracy and latency constraints.

Contribution

GAR is a novel constrained optimization approach that incorporates real-time carbon-aware routing for LLMs, balancing sustainability with performance.

Findings

GAR reduces CO2 emissions significantly across NLP benchmarks.

GAR maintains accuracy and latency within specified service-level objectives.

The online primal-dual algorithm effectively manages rolling carbon budgets.

Abstract

The growing deployment of large language models (LLMs) makes per-request routing essential for balancing response quality and computational cost across heterogeneous model pools. Current routing methods rarely consider sustainable energy use and CO2 emissions as optimization objectives, despite grid carbon intensity varying by time and region, and models differing significantly in energy consumption. To address this gap, we introduce Green-Aware Routing (GAR), a constrained multi-objective optimization framework that minimizes per-request CO2 emissions subject to explicit accuracy floors and p95-latency service-level objectives (SLOs). GAR employs adaptive constraint optimization through per-dataset floor tuning and incorporates lightweight estimators for correctness, tail latency, and carbon emissions, enabling real-time routing decisions without additional inference passes. We present GAR-PD, a practical online primal-dual routing algorithm for rolling carbon budgets, alongside heuristic variants that achieve high feasibility coverage while limiting accuracy degradation. Comprehensive experiments across standard NLP benchmarks with heterogeneous LLM pools (7B-70B) demonstrate that GAR achieves substantial carbon reductions while maintaining competitive accuracy and p95 latency guarantees, providing a practical, theoretically grounded approach to sustainable LLM inference.