Evaluation of Agents under Simulated AI Marketplace Dynamics

TL;DR

This paper introduces a simulation-based evaluation framework for AI systems operating in competitive marketplaces, capturing dynamics like user switching and market share beyond static benchmarks.

Contribution

It presents Marketplace Evaluation, a novel simulation paradigm that assesses AI systems in competitive, dynamic environments, extending traditional static evaluation methods.

Findings

Enables longitudinal assessment of AI systems in simulated marketplaces.

Provides marketplace-level metrics such as retention and market share.

Formalizes a research agenda for marketplace simulation and evaluation.

Abstract

Modern information access ecosystems consist of mixtures of systems, such as retrieval systems and large language models, and increasingly rely on marketplaces to mediate access to models, tools, and data, making competition between systems inherent to deployment. In such settings, outcomes are shaped not only by benchmark quality but also by competitive pressure, including user switching, routing decisions, and operational constraints. Yet evaluation is still largely conducted on static benchmarks with accuracy-focused measures that assume systems operate in isolation. This mismatch makes it difficult to predict post-deployment success and obscures competitive effects such as early-adoption advantages and market dominance. We introduce Marketplace Evaluation, a simulation-based paradigm that evaluates information access systems as participants in a competitive marketplace. By simulating repeated interactions and evolving user and agent preferences, the framework enables longitudinal evaluation and marketplace-level metrics, such as retention and market share, that complement and can extend beyond traditional accuracy-based metrics. We formalize the framework and outline a research agenda, motivated by business and economics, around marketplace simulation, metrics, optimization, and adoption in evaluation campaigns like TREC.