The Energy Cost of Reasoning: Analyzing Energy Usage in LLMs with Test-time Compute

TL;DR

This paper investigates how test-time compute (TTC) can improve the energy efficiency of large language models by allocating resources during inference, leading to better accuracy-energy trade-offs especially in complex reasoning tasks.

Contribution

It demonstrates that TTC can outperform traditional model scaling in energy efficiency and accuracy, and highlights the importance of adjusting compute based on query complexity.

Findings

TTC surpasses traditional scaling in accuracy/energy efficiency.

Adjusting compute based on output length improves efficiency.

TTC is especially beneficial for complex reasoning tasks.

Abstract

Scaling large language models (LLMs) has driven significant advancements, yet it faces diminishing returns and escalating energy demands. This work explores how test-time compute (TTC) can serve as an energy-efficient complement to conventional scaling strategies by allocating additional computational resources at inference time rather than during training. Specifically, we investigate whether employing TTC can achieve superior accuracy-energy trade-offs compared to simply increasing model size. Our empirical analysis reveals that TTC surpasses traditional model scaling in accuracy/energy efficiency, with notable gains in tasks demanding complex reasoning rather than mere factual recall. Further, we identify a critical interaction between TTC performance and output sequence length, demonstrating that strategically adjusting compute resources at inference time according to query complexity can substantially enhance efficiency. Our findings advocate for TTC as a promising direction, enabling more sustainable, accurate, and adaptable deployment of future language models.