Robust and Diverse Multi-Agent Learning via Rational Policy Gradient

TL;DR

This paper introduces Rational Policy Gradient, a new adversarial optimization method that maintains agent rationality in cooperative multi-agent settings, enabling robust, diverse, and adaptable policies beyond zero-sum scenarios.

Contribution

The paper proposes Rationality-preserving Policy Optimization (RPO) and Rational Policy Gradient (RPG), novel algorithms that prevent self-sabotage in cooperative multi-agent adversarial training.

Findings

RPG extends adversarial algorithms to cooperative settings.

Our approach improves robustness and diversity of policies.

Empirical results show strong performance in various environments.

Abstract

Adversarial optimization algorithms that explicitly search for flaws in agents' policies have been successfully applied to finding robust and diverse policies in multi-agent settings. However, the success of adversarial optimization has been largely limited to zero-sum settings because its naive application in cooperative settings leads to a critical failure mode: agents are irrationally incentivized to self-sabotage, blocking the completion of tasks and halting further learning. To address this, we introduce Rationality-preserving Policy Optimization (RPO), a formalism for adversarial optimization that avoids self-sabotage by ensuring agents remain rational--that is, their policies are optimal with respect to some possible partner policy. To solve RPO, we develop Rational Policy Gradient (RPG), which trains agents to maximize their own reward in a modified version of the original game in which we use opponent shaping techniques to optimize the adversarial objective. RPG enables us to extend a variety of existing adversarial optimization algorithms that, no longer subject to the limitations of self-sabotage, can find adversarial examples, improve robustness and adaptability, and learn diverse policies. We empirically validate that our approach achieves strong performance in several popular cooperative and general-sum environments. Our project page can be found at https://rational-policy-gradient.github.io.