Sustainable Carbon-Aware and Water-Efficient LLM Scheduling in Geo-Distributed Cloud Datacenters

TL;DR

This paper introduces SLIT, a machine learning-based framework for optimizing the sustainability of LLM inference in geo-distributed cloud datacenters by balancing quality, carbon footprint, water use, and energy costs.

Contribution

It presents a novel metaheuristic framework that co-optimizes multiple sustainability metrics for LLM inference across distributed datacenters, addressing environmental concerns.

Findings

Reduces carbon emissions and water usage during LLM inference

Improves quality of service while maintaining sustainability goals

Demonstrates effectiveness through simulation results

Abstract

In recent years, Large Language Models (LLM) such as ChatGPT, CoPilot, and Gemini have been widely adopted in different areas. As the use of LLMs continues to grow, many efforts have focused on reducing the massive training overheads of these models. But it is the environmental impact of handling user requests to LLMs that is increasingly becoming a concern. Recent studies estimate that the costs of operating LLMs in their inference phase can exceed training costs by 25x per year. As LLMs are queried incessantly, the cumulative carbon footprint for the operational phase has been shown to far exceed the footprint during the training phase. Further, estimates indicate that 500 ml of fresh water is expended for every 20-50 requests to LLMs during inference. To address these important sustainability issues with LLMs, we propose a novel framework called SLIT to co-optimize LLM quality of service (time-to-first token), carbon emissions, water usage, and energy costs. The framework utilizes a machine learning (ML) based metaheuristic to enhance the sustainability of LLM hosting across geo-distributed cloud datacenters. Such a framework will become increasingly vital as LLMs proliferate.