Beyond Test-Time Compute Strategies: Advocating Energy-per-Token in LLM Inference

TL;DR

This paper investigates the energy-accuracy trade-offs in large and small language models during inference, proposing energy-per-token as a new metric and dynamic reasoning control to promote sustainable AI practices.

Contribution

It introduces energy-per-token as a novel metric and advocates for energy-aware inference strategies, including dynamic reasoning control, to optimize sustainability in LLM deployment.

Findings

Energy-per-Token effectively measures inference energy efficiency.

Dynamic reasoning control balances accuracy and energy consumption.

Transformer input-output token dynamics influence energy consumption patterns.

Abstract

Large Language Models (LLMs) demonstrate exceptional performance across diverse tasks but come with substantial energy and computational costs, particularly in request-heavy scenarios. In many real-world applications, the full scale and capabilities of LLMs are often unnecessary, as Small Language Models (SLMs) can provide accurate responses for simpler text generation tasks. When enhanced with advanced reasoning strategies, such as Chain-of-Thought (CoT) prompting or Majority Voting, SLMs can approach the performance of larger models while reducing overall computational requirements. However, these strategies can also introduce additional energy costs, creating an energy-accuracy trade-off. Our analysis examines these trade-offs in test-time compute strategies for smaller models compared to larger ones, using the MMLU benchmark. Additionally, we explore the input-output token dynamics of transformer architectures, which result in nonlinear hardware energy operation curves for LLMs. To bridge AI research with its physical impact, we propose \textit{energy efficiency metrics}, including Energy-per-Token, as complements to traditional accuracy benchmarks. Beyond model selection, we propose controlled reasoning in CoT token generation, using operating curves to regulate reasoning depth dynamically. This vision integrates a energy-aware routing mechanism, ensuring that model selection and inference strategies balance accuracy for sustainable AI deployment.