Evaluating Collective Behaviour of Hundreds of LLM Agents

TL;DR

This paper presents an evaluation framework for analyzing the collective behavior of hundreds of LLM-based agents in social dilemmas, revealing risks of poor societal outcomes and convergence to suboptimal equilibria.

Contribution

It introduces a scalable evaluation method for large populations of LLM agents and demonstrates the societal risks associated with their collective behavior.

Findings

Recent models perform worse in societal outcomes than older models.

Large populations tend to converge to poor societal equilibria.

Diminished benefits of cooperation increase societal risks.

Abstract

As autonomous agents powered by LLM are increasingly deployed in society, understanding their collective behaviour in social dilemmas becomes critical. We introduce an evaluation framework where LLMs generate strategies encoded as algorithms, enabling inspection prior to deployment and scaling to populations of hundreds of agents -- substantially larger than in previous work. We find that more recent models tend to produce worse societal outcomes compared to older models when agents prioritise individual gain over collective benefits. Using cultural evolution to model user selection of agents, our simulations reveal a significant risk of convergence to poor societal equilibria, particularly when the relative benefit of cooperation diminishes and population sizes increase. We release our code as an evaluation suite for developers to assess the emergent collective behaviour of their models.