Prompt Optimization Enables Stable Algorithmic Collusion in LLM Agents

TL;DR

This paper demonstrates that prompt optimization can lead LLM agents to develop stable collusive strategies in market simulations, raising AI safety concerns.

Contribution

It introduces a meta-learning framework for prompt optimization that enables emergent collusive behaviors in autonomous LLM agents.

Findings

Meta-prompt optimization improves coordination among agents.

Agents discover stable collusion strategies that generalize to new markets.

Analysis reveals systematic mechanisms for coordination through shared strategies.

Abstract

LLM agents in markets present algorithmic collusion risks. While prior work shows LLM agents reach supracompetitive prices through tacit coordination, existing research focuses on hand-crafted prompts. The emerging paradigm of prompt optimization necessitates new methodologies for understanding autonomous agent behavior. We investigate whether prompt optimization leads to emergent collusive behaviors in market simulations. We propose a meta-learning loop where LLM agents participate in duopoly markets and an LLM meta-optimizer iteratively refines shared strategic guidance. Our experiments reveal that meta-prompt optimization enables agents to discover stable tacit collusion strategies with substantially improved coordination quality compared to baseline agents. These behaviors generalize to held-out test markets, indicating discovery of general coordination principles. Analysis of evolved prompts reveals systematic coordination mechanisms through stable shared strategies. Our findings call for further investigation into AI safety implications in autonomous multi-agent systems.