AGFT: An Adaptive GPU Frequency Tuner for Real-Time LLM Inference Optimization

TL;DR

AGFT is an adaptive GPU frequency tuning framework using reinforcement learning to optimize energy efficiency during real-time LLM inference, reducing energy consumption by over 40% with minimal latency impact.

Contribution

We introduce AGFT, a novel reinforcement learning-based framework that dynamically adjusts GPU frequencies for energy-efficient LLM inference without performance loss.

Findings

44.3% GPU energy savings achieved

Less than 10% latency overhead

Up to 40.3% energy-delay product improvement

Abstract

The explosive growth of interactive Large Language Models (LLMs) has placed unprecedented demands for low latency on cloud GPUs, forcing them into high-power modes and causing escalating energy costs. Real-time inference workloads exhibit significant dynamic volatility, presenting substantial energy-saving opportunities. However, traditional static or rule-based power management strategies struggle to exploit these opportunities without compromising peak performance. To address this challenge, we propose AGFT (An Adaptive GPU Frequency Tuner), a framework that employs online reinforcement learning to autonomously learn an optimal frequency tuning policy. By monitoring real-time features like request load and latency, AGFT utilizes fine-grained frequency control for precise adjustments and intelligent action space pruning for stable, efficient decision-making. This creates a robust, automated energy management solution. We comprehensively evaluated AGFT in an environment simulating realistic, fluctuating inference requests. The experimental results demonstrate that AGFT successfully saves 44.3% of GPU energy consumption while introducing a minimal performance latency overhead of under 10%. This achievement translates into a comprehensive Energy-Delay Product (EDP) optimization of up to 40.3%, clearly showing that our framework can significantly enhance the energy efficiency and economic benefits of existing LLM inference clusters without compromising service quality.