Characterising Simulation-Based Program Equilibria

TL;DR

This paper extends the concept of simulation-based program equilibria to more players and scenarios, proving a folk theorem and characterizing equilibria with and without shared randomness, thus broadening the understanding of robust AI agent interactions.

Contribution

We generalize Oesterheld's $ ext{ extsterling}$Grounded$ ext{ extsterling}$ ext{ extsterling}$ ext{ extsterling}$Bot to more players and settings, proving a folk theorem and expanding the range of equilibria achievable.

Findings

Proved a folk theorem for simulation-based programs with shared randomness.

Characterized equilibria in settings without shared randomness.

Demonstrated limitations of simulation-based programs in replicating Tennenholtz's folk theorem.

Abstract

In Tennenholtz's program equilibrium, players of a game submit programs to play on their behalf. Each program receives the other programs' source code and outputs an action. This can model interactions involving AI agents, mutually transparent institutions, or commitments. Tennenholtz (2004) proves a folk theorem for program games, but the equilibria constructed are very brittle. We therefore consider simulation-based programs -- i.e., programs that work by running opponents' programs. These are relatively robust (in particular, two programs that act the same are treated the same) and are more practical than proof-based approaches. Oesterheld's (2019) $\epsilon$Grounded$\pi$Bot is such an approach. Unfortunately, it is not generally applicable to games of three or more players, and only allows for a limited range of equilibria in two player games. In this paper, we propose a generalisation to Oesterheld's (2019) $\epsilon$Grounded$\pi$Bot. We prove a folk theorem for our programs in a setting with access to a shared source of randomness. We then characterise their equilibria in a setting without shared randomness. Both with and without shared randomness, we achieve a much wider range of equilibria than Oesterheld's (2019) $\epsilon$Grounded$\pi$Bot. Finally, we explore the limits of simulation-based program equilibrium, showing that the Tennenholtz folk theorem cannot be attained by simulation-based programs without access to shared randomness.