SAVOIR: Learning Social Savoir-Faire via Shapley-based Reward Attribution

TL;DR

SAVOIR introduces a Shapley-based framework for training social language agents, improving credit assignment in multi-turn dialogues by evaluating strategic potential and ensuring fair reward distribution.

Contribution

The paper presents a novel cooperative game theory approach combining prospective valuation and Shapley values for social RL, achieving state-of-the-art results.

Findings

SAVOIR outperforms existing methods on the SOTOPIA benchmark.

A 7B model with SAVOIR matches or exceeds proprietary models like GPT-4o.

Large reasoning models underperform in social intelligence tasks.

Abstract

Social intelligence, the ability to navigate complex interpersonal interactions, presents a fundamental challenge for language agents. Training such agents via reinforcement learning requires solving the credit assignment problem: determining how individual utterances contribute to multi-turn dialogue outcomes. Existing approaches directly employ language models to distribute episode-level rewards, yielding attributions that are retrospective and lack theoretical grounding. We propose SAVOIR (ShApley Value fOr SocIal RL), a novel principled framework grounded in cooperative game theory. Our approach combines two complementary principles: expected utility shifts evaluation from retrospective attribution to prospective valuation, capturing an utterance's strategic potential for enabling favorable future trajectories; Shapley values ensure fair credit distribution with axiomatic guarantees of efficiency, symmetry, and marginality. Experiments on the SOTOPIA benchmark demonstrate that SAVOIR achieves new state-of-the-art performance across all evaluation settings, with our 7B model matching or exceeding proprietary models including GPT-4o and Claude-3.5-Sonnet. Notably, even large reasoning models consistently underperform, suggesting social intelligence requires qualitatively different capabilities than analytical reasoning.