Sustaining Cooperation in Populations Guided by AI: A Folk Theorem for LLMs

TL;DR

This paper explores how shared guidance from large language models (LLMs) can sustain cooperation among agents with misaligned incentives through a novel folk theorem for repeated interactions.

Contribution

It introduces a new folk theorem for LLM-guided populations, demonstrating the potential for sustained cooperation despite indirect observation and strategic complexity.

Findings

Shared LLM guidance can induce cooperation in one-shot interactions when influencing multiple roles.

A folk theorem for repeated LLM-guided interactions shows all feasible outcomes can be sustained as equilibria.

The results highlight the role of shared AI instructions in aligning strategic behavior among agents.

Abstract

Large language models (LLMs) are increasingly used to provide instructions to many agents who interact with one another. Such shared reliance couples agents who appear to act independently: they may in fact be guided by a common model. This coupling can change the prospects for cooperation among agents with misaligned incentives. We study settings in which multiple LLMs each advise a population of clients who participate in instances of an underlying game, creating strategic interaction at the level of the LLMs themselves. This induces a meta-game among the LLMs, mediated through clients. We first analyze the one-shot setting, where shared instructions can change equilibrium behavior only when an LLM may influence more than one role in the same interaction; in such cases, cooperation may emerge, and the effect of client share can be beneficial, harmful, or non-monotone, depending on the base game. Our main result concerns the repeated setting. We prove a folk theorem for LLMs: despite indirect observation and the clients' inability to identify which LLM advised their opponents, all feasible and individually rational outcomes can be sustained as $\varepsilon$-equilibria. The result does not follow from the standard folk theorem and requires new proof techniques. Together, these results show that shared LLM guidance can sustain cooperation among populations of agents even when the underlying incentives are misaligned.