Learning Robust Social Strategies with Large Language Models

TL;DR

This paper investigates multi-agent social dilemmas with large language models, demonstrating that standard reinforcement learning leads to self-interested behavior, and proposes an opponent-aware training method to promote cooperation and robustness.

Contribution

It introduces Advantage Alignment, an opponent-learning awareness algorithm adapted for LLMs, and a new social dilemma environment, Trust-and-Split, to improve multi-agent cooperation.

Findings

Advantage Alignment increases collective payoffs in social dilemmas.

RL-trained LLMs tend to develop opportunistic, exploitative behaviors.

The proposed method enhances robustness against greedy exploitation.

Abstract

As agentic AI becomes more widespread, agents with distinct and possibly conflicting goals will interact in complex ways. These multi-agent interactions pose a fundamental challenge, particularly in social dilemmas, where agents' individual incentives can undermine collective welfare. While reinforcement learning (RL) has been effective for aligning large language models (LLMs) in the single-agent regime, prior small-network results suggest that standard RL in multi-agent settings often converges to defecting, self-interested policies. We show the same effect in LLMs: despite cooperative priors, RL-trained LLM agents develop opportunistic behavior that can exploit even advanced closed-source models. To address this tendency of RL to converge to poor equilibria, we adapt a recent opponent-learning awareness algorithm, Advantage Alignment, to fine-tune LLMs toward multi-agent cooperation and non-exploitability. We then introduce a group-relative baseline that simplifies advantage computation in iterated games, enabling multi-agent training at LLM scale. We also contribute a novel social dilemma environment, Trust-and-Split, which requires natural language communication to achieve high collective welfare. Across a wide range of social dilemmas, policies learned with Advantage Alignment achieve higher collective payoffs while remaining robust against exploitation by greedy agents. We release all of our code to support future work on multi-agent RL training for LLMs.