Multi-agent cooperation through in-context co-player inference

TL;DR

This paper demonstrates that sequence models can naturally learn to cooperate in multi-agent settings by in-context adaptation to diverse co-players, without hardcoded assumptions or explicit timescale separation.

Contribution

It shows that in-context learning in sequence models enables natural co-player adaptation and cooperation without relying on predefined learning rules or timescale distinctions.

Findings

In-context learning induces best-response strategies in multi-agent interactions.

Vulnerability to extortion naturally leads to mutual shaping and cooperation.

Diverse co-player training promotes scalable cooperative behavior.

Abstract

Achieving cooperation among self-interested agents remains a fundamental challenge in multi-agent reinforcement learning. Recent work showed that mutual cooperation can be induced between "learning-aware" agents that account for and shape the learning dynamics of their co-players. However, existing approaches typically rely on hardcoded, often inconsistent, assumptions about co-player learning rules or enforce a strict separation between "naive learners" updating on fast timescales and "meta-learners" observing these updates. Here, we demonstrate that the in-context learning capabilities of sequence models allow for co-player learning awareness without requiring hardcoded assumptions or explicit timescale separation. We show that training sequence model agents against a diverse distribution of co-players naturally induces in-context best-response strategies, effectively functioning as learning algorithms on the fast intra-episode timescale. We find that the cooperative mechanism identified in prior work-where vulnerability to extortion drives mutual shaping-emerges naturally in this setting: in-context adaptation renders agents vulnerable to extortion, and the resulting mutual pressure to shape the opponent's in-context learning dynamics resolves into the learning of cooperative behavior. Our results suggest that standard decentralized reinforcement learning on sequence models combined with co-player diversity provides a scalable path to learning cooperative behaviors.