Fair Cooperation in Mixed-Motive Games via Conflict-Aware Gradient Adjustment

TL;DR

This paper introduces a conflict-aware gradient adjustment method for multi-agent reinforcement learning that promotes cooperation and fairness among agents in mixed-motive environments, outperforming existing approaches.

Contribution

It presents a novel adaptive gradient adjustment technique that explicitly resolves conflicts between individual and collective objectives, ensuring fairness and improving social welfare.

Findings

Outperforms baselines in social welfare metrics

Guarantees monotonic improvement in fairness and collective goals

Demonstrates effectiveness in sequential social dilemma environments

Abstract

Multi-agent reinforcement learning in mixed-motive settings presents a fundamental challenge: agents must balance individual interests with collective goals, which are neither fully aligned nor strictly opposed. To address this, reward restructuring methods such as gifting and intrinsic motivation have been proposed. However, these approaches primarily focus on promoting cooperation by managing the trade-off between individual and collective returns, without explicitly addressing fairness with respect to the agents' task-specific rewards. In this paper, we propose an adaptive conflict-aware gradient adjustment method that promotes cooperation while ensuring fairness in individual rewards. The proposed method dynamically balances policy gradients derived from individual and collective objectives in situations where the two objectives are in conflict. By explicitly resolving such conflicts, our method improves collective performance while preserving fairness across agents. We provide theoretical results that guarantee monotonic non-decreasing improvement in both the collective and individual objectives and ensure fairness. Empirical results in sequential social dilemma environments demonstrate that our approach outperforms baselines in terms of social welfare while ensuring fairness among agents.