Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning

TL;DR

This paper introduces a probabilistic recursive reasoning framework for multi-agent reinforcement learning, enabling agents to model and respond to opponents' beliefs about their own future actions, improving convergence in complex games.

Contribution

It proposes a novel probabilistic recursive reasoning approach using variational Bayes, with algorithms that converge to Nash equilibria in multi-agent settings.

Findings

PR2 algorithms outperform gradient-based methods in complex games.

Reasoning about opponents' beliefs enhances multi-agent learning stability.

The framework successfully converges in non-trivial equilibrium scenarios.

Abstract

Humans are capable of attributing latent mental contents such as beliefs or intentions to others. The social skill is critical in daily life for reasoning about the potential consequences of others' behaviors so as to plan ahead. It is known that humans use such reasoning ability recursively by considering what others believe about their own beliefs. In this paper, we start from level-$1$ recursion and introduce a probabilistic recursive reasoning (PR2) framework for multi-agent reinforcement learning. Our hypothesis is that it is beneficial for each agent to account for how the opponents would react to its future behaviors. Under the PR2 framework, we adopt variational Bayes methods to approximate the opponents' conditional policies, to which each agent finds the best response and then improve their own policies. We develop decentralized-training-decentralized-execution algorithms, namely PR2-Q and PR2-Actor-Critic, that are proved to converge in the self-play scenarios when there exists one Nash equilibrium. Our methods are tested on both the matrix game and the differential game, which have a non-trivial equilibrium where common gradient-based methods fail to converge. Our experiments show that it is critical to reason about how the opponents believe about what the agent believes. We expect our work to contribute a new idea of modeling the opponents to the multi-agent reinforcement learning community.