Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arena

TL;DR

This paper introduces AucArena, an auction simulation environment to evaluate LLMs' strategic reasoning and planning skills, revealing their strengths and limitations in dynamic, competitive scenarios.

Contribution

The paper presents AucArena, a novel auction-based evaluation suite for testing LLMs' strategic and planning abilities in complex, unpredictable environments.

Findings

LLMs like GPT-4 can manage budgets and goals in auctions

Adaptive strategies improve LLM performance in auction tasks

Simple methods can sometimes outperform complex LLM strategies

Abstract

Recent advancements in Large Language Models (LLMs) showcase advanced reasoning, yet NLP evaluations often depend on static benchmarks. Evaluating this necessitates environments that test strategic reasoning in dynamic, competitive scenarios requiring long-term planning. We introduce AucArena, a novel evaluation suite that simulates auctions, a setting chosen for being highly unpredictable and involving many skills related to resource and risk management, while also being easy to evaluate. We conduct controlled experiments using state-of-the-art LLMs to power bidding agents to benchmark their planning and execution skills. Our research demonstrates that LLMs, such as GPT-4, possess key skills for auction participation, such as budget management and goal adherence, which improve with adaptive strategies. This highlights LLMs' potential in modeling complex social interactions in competitive contexts. However, variability in LLM performance and occasional outperformance by simpler methods indicate opportunities for further advancements in LLM design and the value of our simulation environment for ongoing testing and refinement.