MarketBench: Evaluating AI Agents as Market Participants

TL;DR

MarketBench is a benchmark designed to evaluate AI agents' ability to participate effectively in markets, focusing on their self-assessment accuracy and calibration, demonstrated through experiments with LLMs on a software engineering task set.

Contribution

We introduce MarketBench, a novel benchmark for assessing AI agents' market participation capabilities, highlighting calibration issues and the impact of self-assessment on market coordination.

Findings

LLMs are miscalibrated on success probability and token usage.

Adding prior capability information modestly improves calibration.

Self-assessment is a key bottleneck for market-based AI coordination.

Abstract

Markets are a promising way to coordinate AI agent activity for similar reasons to those used to justify markets more broadly. In order to effectively participate in markets, agents need to have informative signals of their own ability to successfully complete a task and the cost of doing so. We propose MarketBench, a benchmark for assessing whether AI agents have these capabilities. We use a 93-task subset of SWE-bench Lite, a software engineering benchmark, with six recently released LLMs as a demonstration. These LLMs are miscalibrated on both success probability and token usage, and auctions built from these self-reports diverge from a full-information allocation. A follow-up intervention where we add information about capabilities from prior experiments to the context improves calibration, but only modestly narrows the gap to a full-information benchmark. We also document the performance of a market-based scaffolding with these LLMs. Our results point to self-assessment as a key bottleneck for market-style coordination of AI agents.