Promoting Sustainable Web Agents: Benchmarking and Estimating Energy Consumption through Empirical and Theoretical Analysis

TL;DR

This paper investigates the energy consumption of web agents like OpenAI's Operator and Google's Project Mariner through empirical benchmarking and theoretical estimation, emphasizing sustainability concerns and the need for dedicated energy metrics.

Contribution

It provides the first combined empirical and theoretical analysis of web agent energy use, highlighting the impact of design philosophies and advocating for standardized energy metrics.

Findings

Different web agent designs greatly affect energy consumption.

More energy use does not always lead to better performance.

Lack of transparency hampers accurate energy estimation.

Abstract

Web agents, like OpenAI's Operator and Google's Project Mariner, are powerful agentic systems pushing the boundaries of Large Language Models (LLM). They can autonomously interact with the internet at the user's behest, such as navigating websites, filling search masks, and comparing price lists. Though web agent research is thriving, induced sustainability issues remain largely unexplored. To highlight the urgency of this issue, we provide an initial exploration of the energy and $CO_2$ cost associated with web agents from both a theoretical -via estimation- and an empirical perspective -by benchmarking. Our results show how different philosophies in web agent creation can severely impact the associated expended energy, and that more energy consumed does not necessarily equate to better results. We highlight a lack of transparency regarding disclosing model parameters and processes used for some web agents as a limiting factor when estimating energy consumption. Our work contributes towards a change in thinking of how we evaluate web agents, advocating for dedicated metrics measuring energy consumption in benchmarks.