Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models

TL;DR

This paper benchmarks the energy consumption of large language models during inference across various NLP tasks, revealing key factors affecting energy use and proposing strategies for more sustainable deployment.

Contribution

It provides the first comprehensive benchmarking of LLM inference energy across diverse tasks, models, prompts, and system factors, with actionable recommendations for energy efficiency.

Findings

Inference energy correlates strongly with output token length and response time.

Quantization and optimal batch sizes significantly reduce energy consumption.

Targeted prompt phrases can lower energy usage during inference.

Abstract

Large language models (LLMs) are increasingly recognized for their exceptional generative capabilities and versatility across various tasks. However, the high inference costs associated with these models have not received adequate attention, particularly when compared to the focus on training costs in existing research. In response to this gap, our study conducts a comprehensive benchmarking of LLM inference energy across a wide range of NLP tasks, where we analyze the impact of different models, tasks, prompts, and system-related factors on inference energy. Specifically, our experiments reveal several interesting insights, including strong correlation of inference energy with output token length and response time. Also, we find that quantization and optimal batch sizes, along with targeted prompt phrases, can significantly reduce energy usage. This study is the first to thoroughly benchmark LLM inference across such a diverse range of aspects, providing insights and offering several recommendations for improving energy efficiency in model deployment.