One Model, All Roles: Multi-Turn, Multi-Agent Self-Play Reinforcement Learning for Conversational Social Intelligence

TL;DR

This paper presents OMAR, a reinforcement learning framework enabling a single AI model to develop social intelligence through multi-turn, multi-agent self-play, capturing complex social norms and behaviors in conversations.

Contribution

Introducing OMAR, a novel multi-agent self-play reinforcement learning approach that trains a single model to understand and exhibit complex social behaviors in conversations.

Findings

Models develop empathy, persuasion, and compromise skills.

Effective learning of social norms without human supervision.

Demonstrates emergent social intelligence in competitive scenarios.

Abstract

This paper introduces OMAR: One Model, All Roles, a reinforcement learning framework that enables AI to develop social intelligence through multi-turn, multi-agent conversational self-play. Unlike traditional paradigms that rely on static, single-turn optimizations, OMAR allows a single model to role-play all participants in a conversation simultaneously, learning to achieve long-term goals and complex social norms directly from dynamic social interaction. To ensure training stability across long dialogues, we implement a hierarchical advantage estimation that calculates turn-level and token-level advantages. Evaluations in the SOTOPIA social environment and Werewolf strategy games show that our trained models develop fine-grained, emergent social intelligence, such as empathy, persuasion, and compromise seeking, demonstrating the effectiveness of learning collaboration even under competitive scenarios. While we identify practical challenges like reward hacking, our results show that rich social intelligence can emerge without human supervision. We hope this work incentivizes further research on AI social intelligence in group conversations.