This Is Taking Too Long -- Investigating Time as a Proxy for Energy Consumption of LLMs

TL;DR

This paper explores using inference time measurements as a practical proxy to estimate the energy consumption of API-based Large Language Models, addressing transparency issues and aiding users in understanding environmental impacts.

Contribution

It introduces a method to infer energy costs of API-based LLMs through inference time, validated by comparison with actual energy measurements from local models.

Findings

Inference time can accurately predict GPU models used by API-based LLMs.

Time measurements enable estimation of energy consumption for opaque API models.

The approach helps users understand the environmental impact of LLMs.

Abstract

The energy consumption of Large Language Models (LLMs) is raising growing concerns due to their adverse effects on environmental stability and resource use. Yet, these energy costs remain largely opaque to users, especially when models are accessed through an API -- a black box in which all information depends on what providers choose to disclose. In this work, we investigate inference time measurements as a proxy to approximate the associated energy costs of API-based LLMs. We ground our approach by comparing our estimations with actual energy measurements from locally hosted equivalents. Our results show that time measurements allow us to infer GPU models for API-based LLMs, grounding our energy cost estimations. Our work aims to create means for understanding the associated energy costs of API-based LLMs, especially for end users.