The Generative Energy Arena (GEA): Incorporating Energy Awareness in Large Language Model (LLM) Human Evaluations

TL;DR

The paper introduces GEA, a human evaluation arena for LLMs that incorporates energy consumption data, revealing users tend to prefer smaller, energy-efficient models when energy awareness is considered.

Contribution

GEA is a novel evaluation framework that integrates energy consumption information into human assessments of LLMs, addressing scalability and energy-awareness in model evaluation.

Findings

Users prefer smaller, energy-efficient models when energy consumption is highlighted.

Energy-aware evaluations influence model ranking towards less energy-intensive options.

Most users do not perceive additional value in larger models when energy costs are considered.

Abstract

The evaluation of large language models is a complex task, in which several approaches have been proposed. The most common is the use of automated benchmarks in which LLMs have to answer multiple-choice questions of different topics. However, this method has certain limitations, being the most concerning, the poor correlation with the humans. An alternative approach, is to have humans evaluate the LLMs. This poses scalability issues as there is a large and growing number of models to evaluate making it impractical (and costly) to run traditional studies based on recruiting a number of evaluators and having them rank the responses of the models. An alternative approach is the use of public arenas, such as the popular LM arena, on which any user can freely evaluate models on any question and rank the responses of two models. The results are then elaborated into a model ranking. An increasingly important aspect of LLMs is their energy consumption and, therefore, evaluating how energy awareness influences the decisions of humans in selecting a model is of interest. In this paper, we present GEA, the Generative Energy Arena, an arena that incorporates information on the energy consumption of the model in the evaluation process. Preliminary results obtained with GEA are also presented, showing that for most questions, when users are aware of the energy consumption, they favor smaller and more energy efficient models. This suggests that for most user interactions, the extra cost and energy incurred by the more complex and top-performing models do not provide an increase in the perceived quality of the responses that justifies their use.