TokenPowerBench: Benchmarking the Power Consumption of LLM Inference

TL;DR

TokenPowerBench is a novel, lightweight benchmarking tool that enables detailed power consumption analysis of large language model inference, facilitating energy efficiency improvements and sustainability efforts.

Contribution

It introduces the first extensible benchmark for measuring LLM inference power consumption, with a comprehensive measurement layer and configurable parameters.

Findings

Effective power measurement across multiple models and sizes.

Insights into how batch size, context length, and quantization affect energy use.

Open source release for community adoption and benchmarking.

Abstract

Large language model (LLM) services now answer billions of queries per day, and industry reports show that inference, not training, accounts for more than 90% of total power consumption. However, existing benchmarks focus on either training/fine-tuning or performance of inference and provide little support for power consumption measurement and analysis of inference. We introduce TokenPowerBench, the first lightweight and extensible benchmark designed for LLM-inference power consumption studies. The benchmark combines (i) a declarative configuration interface covering model choice, prompt set, and inference engine, (ii) a measurement layer that captures GPU-, node-, and system-level power without specialized power meters, and (iii) a phase-aligned metrics pipeline that attributes energy to the prefill and decode stages of every request. These elements make it straight-forward to explore the power consumed by an LLM inference run; furthermore, by varying batch size, context length, parallelism strategy and quantization, users can quickly assess how each setting affects joules per token and other energy-efficiency metrics. We evaluate TokenPowerBench on four of the most widely used model series (Llama, Falcon, Qwen, and Mistral). Our experiments cover from 1 billion parameters up to the frontier-scale Llama3-405B model. Furthermore, we release TokenPowerBench as open source to help users to measure power consumption, forecast operating expenses, and meet sustainability targets when deploying LLM services.